https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12865 Lustre <p>sanity test_160f fails with ‘mds1: User cl6 not registered’. So far this year, there have been 52 sanity test 160f failures with this error; 36 of those failures are for ARM clients.</p> <p>Looking at the suite_log for a recent failure, <a href="https://testing.whamcloud.com/test_sets/8bedad40-ebd5-11e9-b62b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8bedad40-ebd5-11e9-b62b-52540065bddc</a>, we see that user cl6 is registered and that we are able to manipulate the changelog register prior to the error</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 160f: changelog garbage collect (timestamped users) =================================== 20:27:47 (1570566467) CMD: trevis-49vm2 /usr/sbin/lctl get_param mdd.lustre-MDT0000.changelog_mask -n CMD: trevis-49vm2 /usr/sbin/lctl set_param mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0000.changelog_mask=+hsm CMD: trevis-49vm2 /usr/sbin/lctl --device lustre-MDT0000 changelog_register -n CMD: trevis-49vm3 /usr/sbin/lctl get_param mdd.lustre-MDT0001.changelog_mask -n CMD: trevis-49vm3 /usr/sbin/lctl set_param mdd.lustre-MDT0001.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm CMD: trevis-49vm3 /usr/sbin/lctl --device lustre-MDT0001 changelog_register -n CMD: trevis-49vm2 /usr/sbin/lctl get_param mdd.lustre-MDT0002.changelog_mask -n CMD: trevis-49vm2 /usr/sbin/lctl set_param mdd.lustre-MDT0002.changelog_mask=+hsm mdd.lustre-MDT0002.changelog_mask=+hsm CMD: trevis-49vm2 /usr/sbin/lctl --device lustre-MDT0002 changelog_register -n CMD: trevis-49vm3 /usr/sbin/lctl get_param mdd.lustre-MDT0003.changelog_mask -n CMD: trevis-49vm3 /usr/sbin/lctl set_param mdd.lustre-MDT0003.changelog_mask=+hsm mdd.lustre-MDT0003.changelog_mask=+hsm CMD: trevis-49vm3 /usr/sbin/lctl --device lustre-MDT0003 changelog_register -n Registered 4 changelog users: 'cl6 cl6 cl6 cl6' … mds1: verifying user cl6 clear: 19 + 2 == 21 CMD: trevis-49vm2 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users CMD: trevis-49vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0001.changelog_users CMD: trevis-49vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0001.changelog_users lustre-MDT0001: clear the changelog for cl6 to record #10 CMD: trevis-49vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0001.changelog_users mds2: verifying user cl6 clear: 8 + 2 == 10 CMD: trevis-49vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0001.changelog_users CMD: trevis-49vm2 /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.changelog_users CMD: trevis-49vm2 /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.changelog_users lustre-MDT0002: clear the changelog for cl6 to record #2 CMD: trevis-49vm2 /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.changelog_users mds3: verifying user cl6 clear: 0 + 2 == 2 CMD: trevis-49vm2 /usr/sbin/lctl get_param -n mdd.lustre-MDT0002.changelog_users CMD: trevis-49vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0003.changelog_users CMD: trevis-49vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0003.changelog_users lustre-MDT0003: clear the changelog for cl6 to record #2 CMD: trevis-49vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0003.changelog_users mds4: verifying user cl6 clear: 0 + 2 == 2 CMD: trevis-49vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0003.changelog_users total: 8 create in 0.02 seconds: 453.39 ops/second CMD: trevis-49vm2 ps -e -o comm= | grep chlg_gc_thread pdsh@trevis-79vm17: trevis-49vm2: ssh exited with exit code 1 CMD: trevis-49vm2 ps -e -o comm= | grep chlg_gc_thread pdsh@trevis-79vm17: trevis-49vm2: ssh exited with exit code 1 CMD: trevis-49vm3 ps -e -o comm= | grep chlg_gc_thread pdsh@trevis-79vm17: trevis-49vm3: ssh exited with exit code 1 CMD: trevis-49vm3 ps -e -o comm= | grep chlg_gc_thread pdsh@trevis-79vm17: trevis-49vm3: ssh exited with exit code 1 CMD: trevis-49vm2 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.changelog_users sanity test_160f: @@@@@@ FAIL: mds1: User cl6 not registered </pre> </div></div> <p>There is no indication of a problem in any of the console logs.</p> <p>Logs for other failures are at<br/> <a href="https://testing.whamcloud.com/test_sets/ecc72250-ccfd-11e9-a25b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/ecc72250-ccfd-11e9-a25b-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/8da05bea-cff3-11e9-9fc9-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8da05bea-cff3-11e9-9fc9-52540065bddc</a></p> LU-12865

sanity test 160f fails with ‘mds1: User cl6 not registered’

Bug Minor Resolved Fixed Andreas Dilger James Nunez Tue, 15 Oct 2019 19:07:55 +0000 Mon, 15 Jan 2024 23:29:01 +0000 Sat, 18 Jan 2020 15:01:18 +0000 Lustre 2.13.0 Lustre 2.12.3 Lustre 2.12.4 Lustre 2.12.5 Lustre 2.14.0 Lustre 2.12.6 0 4 <p>It looks like this test failure might be a problem in the test script itself? Looking at the MDT0000 debug log shows that the <tt>cl6</tt> user is deregistered by the GC thread because it is idle for too long:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1570566291.903442:0:27506:0:(mdd_trans.c:192:mdd_chlg_garbage_collect()) lustre-MDD0000: Force deregister of ChangeLog user cl7 idle since more than 35s 1570566291.903509:0:27506:0:(mdd_trans.c:192:mdd_chlg_garbage_collect()) lustre-MDD0000: Force deregister of ChangeLog user cl6 idle since more than 11s </pre> </div></div> <p>This is because the test sets <tt>changelog_max_idle_time=10</tt>, but sleeps for 9s and then does a number of operations that could be slow. In rare cases the test ran too long and the MDS evicted the "good" along with the bad one.</p> <p>Patch to fix the test forthcoming.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36468" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36468</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12865" title="sanity test 160f fails with ‘mds1: User cl6 not registered’" class="issue-link" data-issue-key="LU-12865"><del>LU-12865</del></a> tests: fix sanity 160f to be more robust<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7e2a15299f1180d8b7026f46886c85dbf4dacd7b</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36468/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36468/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12865" title="sanity test 160f fails with ‘mds1: User cl6 not registered’" class="issue-link" data-issue-key="LU-12865"><del>LU-12865</del></a> tests: fix sanity 160f to be more robust<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4b0f0164c6ed761897409186376e9edc989323c9</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38833" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38833</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12865" title="sanity test 160f fails with ‘mds1: User cl6 not registered’" class="issue-link" data-issue-key="LU-12865"><del>LU-12865</del></a> tests: fix sanity 160f to be more robust<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 011b8e3eac372032dd8f09576239e7a05d496251</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38833/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38833/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12865" title="sanity test 160f fails with ‘mds1: User cl6 not registered’" class="issue-link" data-issue-key="LU-12865"><del>LU-12865</del></a> tests: fix sanity 160f to be more robust<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: fdc1320ea1a817182291ec67394a903b2e0a4911</p> Related LU-10680 LU-12871 Development Rank 1|i00o33: Rank (Obsolete) 9223372036854775807 Severity