Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-10401] sanity test_133g: timeout during MDT mount https://jira.whamcloud.com/browse/LU-10401 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;andreas.dilger@intel.com&gt;</p> <p>This issue relates to the following test suite run:</p> <p>Info required for matching: sanity 133g<br/> Info required for matching: sanity 133h</p> LU-10401 sanity test_133g: timeout during MDT mount Bug Minor Resolved Fixed Yang Sheng Maloo Fri, 15 Dec 2017 19:21:05 +0000 Thu, 24 Jun 2021 15:15:24 +0000 Sat, 30 May 2020 09:43:08 +0000 Lustre 2.12.0 Lustre 2.14.0 0 8 <p>+1 on b2_12:<br/> <a href="https://testing.whamcloud.com/test_sets/1b84726a-10d7-11ea-a9d7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1b84726a-10d7-11ea-a9d7-52540065bddc</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/1235bfca-16f4-11ea-98f1-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1235bfca-16f4-11ea-98f1-52540065bddc</a></p> <p>+1 on b2_12:<br/> <a href="https://testing.whamcloud.com/test_sets/98320413-d7e1-4cb1-9c78-20b4d126575f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/98320413-d7e1-4cb1-9c78-20b4d126575f</a></p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38444" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38444</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10401" title="sanity test_133g: timeout during MDT mount" class="issue-link" data-issue-key="LU-10401"><del>LU-10401</del></a> tests: fix error from 'tr -d='<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b18389f3f14d2c88268ec5da15a76295cda30c67</p> <p>There are error messages printed during the test run:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: trevis-64vm4 /usr/sbin/lctl list_param -R '*' | grep '=' | tr -d= | egrep -v 'force_lbug|changelog_mask' | xargs badarea_io trevis-64vm4: tr: invalid option -- '=' trevis-64vm4: Try 'tr --help' for more information. </pre> </div></div> <p>It isn't clear whether this is causing the test to fail, or is something spurious, but running RHEL7.8 is failing sanity test_133g repeatedly.</p> <p>I haven't been able to get RHEL7.8 to pass at this point. </p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38444/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38444/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10401" title="sanity test_133g: timeout during MDT mount" class="issue-link" data-issue-key="LU-10401"><del>LU-10401</del></a> tests: fix error from 'tr -d='<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 52b5f4a5c3fc942f2b6aef9dbed780bd2c2a6798</p> <p>Landed for 2.14</p> <p>It's not clear if this patch will solve the failures, it it <em>might</em>. I'd like to keep this issue open until I can get a passing result from e2fsprogs. Unfortunately, there is no way to test e2fsprogs in autotest against a specific patch, so this had to be landed before I could retry with e2fsprogs. </p> <p>Some examples of recent failures, which are specific to CentOS 7.8 and RHEL 8.0:<br/> <a href="https://testing.whamcloud.com/test_sets/a85aeb21-b186-4e60-990c-cf89bb26d855" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/a85aeb21-b186-4e60-990c-cf89bb26d855</a><br/> <a href="https://testing.whamcloud.com/test_sets/75f7b5a2-05d8-4e7b-8ca7-6253da6a7add" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/75f7b5a2-05d8-4e7b-8ca7-6253da6a7add</a></p> <p>YS, can you please look into why this test is failing. It only has problems with CentOS 7.8 and RHEL 8.1, not the regular CentOS 7.7 runs that are part of review testing. </p> <p>From client log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[10864.205961] ptlrpcd_rcv S 0 7446 2 0x80000080 [10864.206882] Call Trace: [10864.207350] ? __schedule+0x253/0x830 [10864.208002] schedule+0x28/0x70 [10864.208566] schedule_timeout+0x16b/0x390 [10864.209283] ? __next_timer_interrupt+0xc0/0xc0 [10864.210087] ptlrpc_set_wait+0x4ba/0x6e0 [ptlrpc] [10864.210904] ? finish_wait+0x80/0x80 [10864.211568] ptlrpc_queue_wait+0x7e/0x210 [ptlrpc] [10864.212407] fld_client_rpc+0x277/0x580 [fld] [10864.213238] ? cfs_trace_unlock_tcd+0x2e/0x80 [libcfs] [10864.214125] fld_client_lookup+0x254/0x470 [fld] [10864.214952] lmv_fld_lookup+0x8c/0x420 [lmv] [10864.215712] lmv_lock_match+0x7c/0x3f0 [lmv] [10864.216621] ll_have_md_lock+0x169/0x3b0 [lustre] [10864.217447] ? vsnprintf+0x101/0x520 [10864.218092] ll_md_blocking_ast+0x60d/0xbd0 [lustre] [10864.218976] ldlm_cancel_callback+0x7b/0x250 [ptlrpc] [10864.219868] ? ldlm_lock_remove_from_lru_nolock+0x38/0xf0 [ptlrpc] [10864.220929] ldlm_lock_cancel+0x55/0x1c0 [ptlrpc] [10864.221772] ldlm_cli_cancel_list_local+0x8f/0x300 [ptlrpc] [10864.222740] ldlm_replay_locks+0x662/0x850 [ptlrpc] [10864.223611] ptlrpc_import_recovery_state_machine+0x868/0x970 [ptlrpc] [10864.224735] ptlrpc_connect_interpret+0x11f0/0x22d0 [ptlrpc] [10864.225717] ? after_reply+0x8de/0xd30 [ptlrpc] [10864.226519] ptlrpc_check_set+0x50c/0x21f0 [ptlrpc] [10864.227366] ? schedule_timeout+0x173/0x390 [10864.228119] ptlrpcd_check+0x3d5/0x5b0 [ptlrpc] [10864.228953] ptlrpcd+0x3d0/0x4c0 [ptlrpc] [10864.229660] ? finish_wait+0x80/0x80 [10864.230329] ? ptlrpcd_check+0x5b0/0x5b0 [ptlrpc] [10864.231162] kthread+0x112/0x130 [10864.231747] ? kthread_flush_work_fn+0x10/0x10 [10864.232516] ret_from_fork+0x35/0x40 </pre> </div></div> <p>Looks like recovery still not finished.</p> <p>YS, can you please make a patch to increase the timeout, or speed up recovery, so this test will pass consistently. </p> <p>Hi, Andreas,</p> <p>I have working to find the root cause. The more stranger thing is that failover was triggered. Looks like some var was exported wrongly?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cln..Failing mds1 on trevis-19vm4 CMD: trevis-19vm4 grep -c /mnt/lustre-mds1' ' /proc/mounts || true Stopping /mnt/lustre-mds1 (opts:) on trevis-19vm4 CMD: trevis-19vm4 umount -d /mnt/lustre-mds1 CMD: trevis-19vm4 lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' || true </pre> </div></div> <p>Thanks,<br/> YangSheng</p> <p>Yang Sheng (ys@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38567" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38567</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10401" title="sanity test_133g: timeout during MDT mount" class="issue-link" data-issue-key="LU-10401"><del>LU-10401</del></a> test: add parameter to print entry type<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1cd538d7db07365a3edd5ef9af5c260d22a673a4</p> <p>So from stacktrace, This issue should duplicate with <a href="https://jira.whamcloud.com/browse/LU-11761" title="blocked MDT mount and high cpu usage from lodXXXX_recYYYY threads" class="issue-link" data-issue-key="LU-11761"><del>LU-11761</del></a>.</p> <p>Hmm so did the fix for <a href="https://jira.whamcloud.com/browse/LU-11761" title="blocked MDT mount and high cpu usage from lodXXXX_recYYYY threads" class="issue-link" data-issue-key="LU-11761"><del>LU-11761</del></a> (marked as included in 2.12.3) not work then?</p> <p>Looks like it still exist. Further investigating.</p> <p>Yang Sheng (ys@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38679" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38679</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10401" title="sanity test_133g: timeout during MDT mount" class="issue-link" data-issue-key="LU-10401"><del>LU-10401</del></a> test: debug patch<br/> Project: tools/e2fsprogs<br/> Branch: master-lustre<br/> Current Patch Set: 1<br/> Commit: a0a337f38866706829cd0a45b7ec63d97fc4406e</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38567/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38567/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10401" title="sanity test_133g: timeout during MDT mount" class="issue-link" data-issue-key="LU-10401"><del>LU-10401</del></a> tests: add -F so list_param prints entry type<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1c54733894f81e854363fbd2d49c141842f73ae4</p> <p>After the latest patch landing, I'm seeing a couple of places where sanity test_133h is failing:<br/> <a href="https://testing.whamcloud.com/test_sets/0a1bd601-7de7-4031-974e-bc138ca14637" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/0a1bd601-7de7-4031-974e-bc138ca14637</a><br/> <a href="https://testing.whamcloud.com/test_sets/9238d945-2814-403b-8ab4-eb96075b4cd4" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/9238d945-2814-403b-8ab4-eb96075b4cd4</a><br/> <a href="https://testing.whamcloud.com/test_sets/1073e4a6-5e88-410a-8c80-f0cb15585635" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1073e4a6-5e88-410a-8c80-f0cb15585635</a><br/> <a href="https://testing.whamcloud.com/test_sets/0f75133d-b81d-483b-bfc8-2b90699c9fb9" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/0f75133d-b81d-483b-bfc8-2b90699c9fb9</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>files do not end with newline: - </pre> </div></div> <p>It is always '-' as the reported filename, so I suspect there is something wrong with how the test was modified, either reporting an error incorrectly, or not printing out the filename properly when a real error is hit. Two of the above failures were with <tt>review-ldiskfs-arm</tt> and two were on e2fsprogs <tt>tiny</tt> sessions. It isn't yet clear if these are permanent or intermittent failures, though I <em>suspect</em> they are permanent failures for those configurations, as I can't imagine how this test would be racy.</p> <p>It is my fault. Since awk get input from stdin, So the FILENAME is '-'. But i want to know why this test case is exist? Looks like it only check whether output a '\n' char for get_param. Does it make sense? </p> <p>YS, can you please push a patch to fix <tt>FILENAME</tt> so we can see which filename is having the problem.</p> <p>Yang Sheng (ys@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38699" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38699</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10401" title="sanity test_133g: timeout during MDT mount" class="issue-link" data-issue-key="LU-10401"><del>LU-10401</del></a> tests: print correct filename while failed<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: beba18261bb3f7b1f0bd1b18163dc80b8b9414a5</p> <p>From log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> [ 8673.618423] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param osc.lustre-OST0006-osc-ffff8b2112ff4000.rpc_stats | awk -v FS=\v -v RS=\v\v 'END { if(NR&gt;0 &amp;&amp; $NF !~ /.*\n$/) print FILENAME}' [ 8673.637811] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param timeout | awk -v FS=\v -v RS=\v\v 'END { if(NR&gt;0 &amp;&amp; $NF !~ /.*\n$/) print FILENAME}' [ 8673.656550] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param console_backoff | awk -v FS=\v -v RS=\v\v 'END { if(NR&gt;0 &amp;&amp; $NF !~ /.*\n$/) print FILENAME}' [ 8673.675441] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param console_max_delay_centisecs | awk -v FS=\v -v RS=\v\v 'END { if(NR&gt;0 &amp;&amp; $NF !~ /.*\n$/) print FILENAME}' [ 8674.097330] Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity test_133h: @@@@@@ FAIL: files do not end with newline: - [ 8674.515181] Lustre: DEBUG MARKER: sanity test_133h: @@@@@@ FAIL: files do not end with newline: - </pre> </div></div> <p>+1 On Master : <a href="https://testing.whamcloud.com/test_sets/619d15cb-a12b-489b-a085-cb473ad81772" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/619d15cb-a12b-489b-a085-cb473ad81772</a></p> <p>After rebasing with current master, seems that patch-set #32 of my change @ <a href="https://review.whamcloud.com/#/c/35856/32" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/35856/32</a> triggers <a href="https://jira.whamcloud.com/browse/LU-10401" title="sanity test_133g: timeout during MDT mount" class="issue-link" data-issue-key="LU-10401"><del>LU-10401</del></a> almost solid !!!! And according to this weeks stats for sanity/test_133h a lot of others auto-testing sessions are also affected.<br/> Could the priority for this issue be raised or at least sanity/test_133h been skipped ?</p> <p>I literally just hit post on a comment in Gerrit about this. There is a patch - <a href="https://review.whamcloud.com/#/c/38699/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/38699/</a> - that you could try rebasing on top of</p> <p>Peter, thanks for your update. But looking at #38699, it looks as a pure debug patch. And according to the current failure rate of sanity/test_133h it should be easy for Yang Sheng to get the associated debug infos now.</p> <p>Hi， Bruno，</p> <p>The latest 38699 is a fixing patch rather than debug patch. So please rebase your patch on top of it to overcome sanity-133h failure.</p> <p>Thanks,<br/> YangSheng</p> <p>I am just going on that this has been used successfully for a couple of other patches that hit these failures. This patch is in master-next ATM so should land shortly.</p> <p>Oops my bad, Peter and Yang Sheng, I had missed the fix part of #38699. Thanks for your help.</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38699/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38699/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10401" title="sanity test_133g: timeout during MDT mount" class="issue-link" data-issue-key="LU-10401"><del>LU-10401</del></a> procs: print new line based on distro<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0130d7019534611ff05a29fad657226d2eb6b79e</p> <p>Patch has landed to master for 2.14. Patches will need to be rebased if they are failing because of this issue. </p> Duplicate LU-11761 LU-13613 Related LU-9864 LU-8130 LU-13558 LU-7688 LU-14789 Development Rank 1|hzzpmn: Rank (Obsolete) 9223372036854775807 Severity