https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13232 Lustre <p>sanity test_160j fails with 'read changelog failed' for PPC client testing 100% of the time. </p> <p>Looking at a recent failure at <a href="https://testing.whamcloud.com/test_sets/d3720002-4a27-11ea-b69a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d3720002-4a27-11ea-b69a-52540065bddc</a>, the actual error is a problem with the input to cat</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Registered 1 changelog users: 'cl3' total: 2 create in 0.00 seconds: 1052.66 ops/second cat: -: Invalid argument sanity test_160j: @@@@@@ FAIL: read changelog failed Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:6121:error() = /usr/lib64/lustre/tests/sanity.sh:14350:test_160j() </pre> </div></div> <p>The code that is failing in sanity test 160j is</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 14341 # read changelog 14342 cat &lt;&amp;4 &gt;/dev/<span class="code-keyword">null</span> || error <span class="code-quote">"read changelog failed"</span> </pre> </div></div> <p>Looking at the client1 (vm12) console log, we see</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 5314.374481] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ===================== 01:24:59 (1581125099) [ 5314.494530] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre2 [ 5314.506580] Lustre: DEBUG MARKER: mount -t lustre -o user_xattr,flock trevis-10vm12@tcp:/lustre /mnt/lustre2 [ 5314.555637] Lustre: Mounted lustre-client [ 5315.555507] Lustre: 10940:0:(llog_cat.c:808:llog_cat_process_common()) lustre-MDT0000-mdc-c0000000b5687800: invalid record in catalog [0x5:0x0:0xa]:0: rc = -22 [ 5315.555690] LustreError: 10940:0:(mdc_changelog.c:295:chlg_load()) lustre-MDT0000-mdc-c0000000b5687800: fail to process llog: rc = -22 [ 5315.600825] Lustre: Unmounted lustre-client [ 5315.777197] Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity test_160j: @@@@@@ FAIL: read changelog failed </pre> </div></div> <p>sanity test 160j started failing for PPC clients as soon as it was first introduced/landed on 27 SEPT 2019.</p> <p>Logs for more PPC client sanity test 160j failures are at<br/> <a href="https://testing.whamcloud.com/test_sets/717d4832-1dba-11ea-80b4-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/717d4832-1dba-11ea-80b4-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/5e7bd63a-f7af-11e9-b62b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5e7bd63a-f7af-11e9-b62b-52540065bddc</a></p> PPC clients LU-13232

sanity test 160j fails with 'read changelog failed'

Bug Minor Resolved Fixed James Nunez James Nunez ppc Mon, 10 Feb 2020 22:17:01 +0000 Thu, 20 Feb 2020 13:16:06 +0000 Thu, 20 Feb 2020 13:16:06 +0000 Lustre 2.14.0 Lustre 2.14.0 0 3 <p>This looks like it may be the root cause of many later failures. This test unmounts the client, then fails (likely because of unexpected output), then doesn't remount the client again. All of the later failures are because there is no Lustre client mounted. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 160j: client can be umounted while its chanangelog is being used CMD: trevis-77vm7.trevis.whamcloud.com mount -t lustre -o user_xattr,flock trevis-10vm12@tcp:/lustre /mnt/lustre2 : cat: -: Invalid argument sanity test_160j: @@@@@@ FAIL: read changelog failed </pre> </div></div> <p>Looking at the test itself, this is pretty clear:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # umount the first lustre mount umount $MOUNT </pre> </div></div> <p>This should have <tt>stack_trap</tt> calls to undo the various changes in the test, like mount the client, unmount client2, close the file descriptors, etc. rather than doing this manually at the end of the test. I suspect with a simple patch to clean up after this failure that many of the following failures will go away also. </p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37550" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37550</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13232" title="sanity test 160j fails with &#39;read changelog failed&#39;" class="issue-link" data-issue-key="LU-13232"><del>LU-13232</del></a> tests: add stack_trap to clean up snaity 160j<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9bfbb05901029b637a9b12262eeff181b8d70348</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37550/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37550/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13232" title="sanity test 160j fails with &#39;read changelog failed&#39;" class="issue-link" data-issue-key="LU-13232"><del>LU-13232</del></a> tests: add stack_trap to clean up sanity 160j<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4891c873184ad2fc3e90abc769456166998cace3</p> <p>Landed for 2.14</p> Development Rank 1|i00tjr: Rank (Obsolete) 9223372036854775807 Severity