https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16639 Lustre <p>It looks like a patch has somehow resulted in a leak of job_stats on the server that result in a message being printed at unmount time:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 1095.059674] Lustre: Failing over lustre-MDT0000 [ 1095.090215] LustreError: 31568:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items [ 1095.093184] LustreError: 31568:0:(lprocfs_jobstats.c:137:job_stat_exit()) Skipped 20 previous similar messages [ 1095.146954] Lustre: server umount lustre-MDT0000 complete </pre> </div></div> <p>This started around 2022-03-03 but I haven't isolated it to a specific patch yet. </p> LU-16639

job_stat_exit() should not have any items

Bug Minor Resolved Fixed Andreas Dilger Andreas Dilger Mon, 13 Mar 2023 18:58:55 +0000 Fri, 10 Nov 2023 09:31:46 +0000 Wed, 29 Mar 2023 03:31:31 +0000 Lustre 2.16.0 Lustre 2.16.0 0 3 <p><a href="https://50.222.100.39/kibana/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:&#39;2022-02-01T15:00:00.000Z&#39;,to:&#39;2022-03-05T18:00:00.000Z&#39;))&amp;_a=(columns:!(test_set_name,sub_test_name,test_session_group,test_set_link),filters:!(),index:ffc3bad0-3e61-11ec-b0b9-8ba7a9c0d3d3,interval:auto,query:(language:kuery,query:&#39;!!%20console_log%20~%20job_stat_exit%20and%20test_set_name:sanity%20and%20sub_test_name:test_232a&#39;),sort:!(!(sub_test_date,desc)))" class="external-link" target="_blank" rel="nofollow noopener">According to this Kibana search</a>, it looks like the first reports of this <tt>job_stat_exit()</tt> message are actually back on 2022-03-03.</p> <p>That was when 2.15.0 was released, and most patches have parent commit v2_15_0-RC2-2-g94f4e1f517 "<tt><a href="https://jira.whamcloud.com/browse/LU-15512" title="Infinite loop in lnet_discover_peer_locked()" class="issue-link" data-issue-key="LU-15512"><del>LU-15512</del></a> lnet: Stop discovery on deleted peer NI</tt>", which was the only patch that had been landed in a couple of weeks. However, there are a few patches in an LNet series <a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/46653/5" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/46653/5</a> that were run on 2022-03-03 that hit the problem, <b>BUT</b> were based on an earlier parent commit, so this might indicate that the issue is at least partly related to the test environment and not the patches themselves.</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50283" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50283</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16639" title="job_stat_exit() should not have any items" class="issue-link" data-issue-key="LU-16639"><del>LU-16639</del></a> misc: improve console error messages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c50c6467958f4287fbcc3abec84e345b9cbf4c24</p> <p>I've pushed the above patch which improves the <tt>job_stats_exit()</tt> message to print the <tt>jn_name</tt> field.</p> <p>In theory, the <tt>lprocfs_job_cleanup()</tt> function should have freed all of the job stats before the <tt>cfs_hash_putref()<del>&gt;cfs_hash_destroy()</del>&gt;job_stat_exit()</tt> path cleans up the remaining stats. It looks like the stats are freed by calling <tt>cfs_hash_put_locked()-&gt;job_free()</tt>.</p> <p>I'm not sure if it is related, but there was an autotest patch landed just before the problem appeared that could potentially be related:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit dbf96c013a17af99b9968f7c79e49f949c33e881 Author: Charlie Olmstead &lt;charlie@whamcloud.com&gt; CommitDate: Tue Mar 1 17:58:17 2022 +0000 ATM-2317 - Monitor test framework process Added logic to monitor the status of the test framework by the presence of &lt;shared_log_dir/run_test.sh.pid. The presence of that file signals to AT that the test framework is running. This allows AT to monitor the status of the framework without logging into the test controller every X seconds. The pid file is deleted when TestStep.runtestscript exits. because of the newly created wrapper. The wrapper has 2 functions: - Execute TestStep.runtestscript in a process group so it and all sub-processes can be killed at once by AT - Remove &lt;shared_log_dir&gt;/run_test.sh.pid file when TestStep.runtestscript exits In addition to the monitoring I've ported over the dynamic execute/rexec logic from loadjenkinsbuild. It allows callers to dynamically build execution commands by changing the method name called: e.g. rexec_no_retry, rexec_no_retry_data, rexec_status_no_retry!, etc. Also: - Removed the unnecessary reboot_without_install? check in the provisioner. This is a relic of old-school provisioning - Improved selinux/firewall enabling/disabling logic - Paritioner: removed logic that runs 'ls /dev' if pvcreate fails. I looked through all of the logs and it has not happened nor have we had any sessions that have failed with 'pvcreate failed, unable to continue' since I can remember. The rest of the changes to this file are shortening of long lines. Change-Id: I5f960bf5210e90a37aa66cdac7a3d260632a5279 Reviewed-on: https://review.whamcloud.com/46528 </pre> </div></div> <p>I did bisection:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> COMMIT TESTED PASSED FAILED COMMIT DESCRIPTION 9cf4dddd52 1 0 1 BAD LU-14831 osd-ldiskfs: uninited osd_inode_id is used 3a9176d5bb 1 0 1 BAD LU-14174 lfs: llapi_mirror_find() signed <span class="code-keyword">return</span> 3a83078628 1 0 1 BAD LU-14409 ldiskfs: Add support <span class="code-keyword">for</span> SUSE 5.3.18-24.46.1 c83304607a 8 0 8 BAD LU-12391 tests: mdsrate tests improvements ea2cd3af7b 8 0 8 BAD LU-11407 obdclass: add start time to stats files 9a5bace55a 10 10 0 GOOD LU-15115 ptlrpc: recalc timer on EINPROGRESS reply 3038917f12 10 10 0 GOOD LU-2084 lnet: don't retry allocating router buffers </pre> </div></div> <p>Thanks Alex. It looks like I messed up the final job stats cleanup in <tt>lprocfs_job_cleanup()</tt> in that patch. </p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50283/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50283/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16639" title="job_stat_exit() should not have any items" class="issue-link" data-issue-key="LU-16639"><del>LU-16639</del></a> misc: cleanup concole messages<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8f40a3d7110da1af8e310a4b7f40b86f13080938</p> <p>Landed for 2.16</p> Related LU-16205 Development Rank 1|i03g6f: Rank (Obsolete) 9223372036854775807 Severity