https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13205 Lustre <p>Looking at a recent failure, <a href="https://testing.whamcloud.com/test_sets/52073ba0-4715-11ea-b69a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/52073ba0-4715-11ea-b69a-52540065bddc</a>, sanity-pfl test 16a fails with the following in the client test_log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity-pfl test 16a: Verify setstripe/getstripe with YAML config file ============================= 05:25:02 (1580793902) CMD: trevis-4vm12 dumpe2fs -h /dev/mapper/mds1_flakey 2&gt;&amp;1 | grep -E -q '(ea_inode|large_xattr)' 1. PFL file getstripe --yaml /mnt/lustre/d16a.sanity-pfl/f16a.sanity-pfl setstripe --yaml=/mnt/lustre/d16a.sanity-pfl/template /mnt/lustre/d16a.sanity-pfl/f16a.sanity-pfl.copy Set stripe size 4096 failed: Invalid argument lfs setstripe: cannot build layout from YAML file /mnt/lustre/d16a.sanity-pfl/template. error: setstripe: can't create composite layout from template file /mnt/lustre/d16a.sanity-pfl/template sanity-pfl test_16a: @@@@@@ FAIL: setstripe /mnt/lustre/d16a.sanity-pfl/f16a.sanity-pfl.copy failed Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:6121:error() = /usr/lib64/lustre/tests/test-framework.sh:9710:verify_yaml_layout() = /usr/lib64/lustre/tests/sanity-pfl.sh:761:test_16a() </pre> </div></div> <p>The only hint of a problem in the console logs is for client 2 (vm9) where we see</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 1212.789487] LustreError: 1758:0:(pack_generic.c:2447:lustre_swab_lov_comp_md_v1()) Invalid magic 0x1 </pre> </div></div> <p>We’ve seen this same error message in sanity-pfl test 14 failures in <a href="https://jira.whamcloud.com/browse/LU-13186" title="sanity-pfl test 14 fails with &#39;/mnt/lustre/d14.sanity-pfl/f14.sanity-pfl: component 4 doesn&#39;t have poolname pool2&#39;" class="issue-link" data-issue-key="LU-13186">LU-13186</a>.</p> <p>Patch <a href="https://review.whamcloud.com/#/c/28425/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28425/</a> for <a href="https://jira.whamcloud.com/browse/LU-9846" title="Overstriping - more than stripe per OST per component" class="issue-link" data-issue-key="LU-9846"><del>LU-9846</del></a> landed to Lustre 2.12.54 on 01 JUNE 2019 and was not back ported to b2_12. This patch moved test_16 to test_16a and created a new test 16b.</p> <p>sanity-pfl test 16a started failing with this error message on 30 JULY 2019 with Lustre 2.12.56.72 and test 16 started failing for b2_12 on 13 AUG 2019 for Lustre 2.12.2.115.</p> <p>sanity-pfl test 16a fails only for PPC clients and fails 100% of the time for PPC. </p> <p>Logs for recent failures are at<br/> <a href="https://testing.whamcloud.com/test_sets/fd407770-4706-11ea-a1c8-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/fd407770-4706-11ea-a1c8-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/07588ab8-2592-11ea-80b4-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/07588ab8-2592-11ea-80b4-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/a2a89ace-1fdb-11ea-adca-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/a2a89ace-1fdb-11ea-adca-52540065bddc</a></p> PPC clients LU-13205

sanity-pfl test 16a fails with “setstripe /mnt/lustre/d16.sanity-pfl/f16.sanity-pfl.copy failed“

Bug Minor Open Unresolved WC Triage James Nunez always_except ppc Wed, 5 Feb 2020 19:03:06 +0000 Mon, 24 Feb 2020 02:25:31 +0000 Lustre 2.13.0 Lustre 2.14.0 Lustre 2.12.4 0 4 <p>Based on the date of the start of failures and the area affected, it looks like this was very likely caused by the landing of:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 9d17996766e0fa93b1029d2422d45d087edde389 CommitDate: Sat Jul 27 00:21:20 2019 +0000 LU-10100 llite: swab LOV EA user data Many sub-tests failed with "Invalid argument" failures on PPC client because of the endianness issue. This patch fixes the issue by adding a common function lustre_swab_lov_user_md() to swab the LOV EA user data. Test-Parameters: clientarch=ppc64 envdefinitions=ONLY=27 testlist=sanity Reviewed-on: https://review.whamcloud.com/35291 </pre> </div></div> <p>It seems that "caused" might be a strong word. Prior to the landing of the <a href="https://jira.whamcloud.com/browse/LU-10100" title="sanity test_27a: setstripe failed with &quot;error on ioctl 0x8008669a for &#39;*&#39; (3): Invalid argument&quot;" class="issue-link" data-issue-key="LU-10100"><del>LU-10100</del></a> patch, sanity-pfl just failed every test. After the landing of <a href="https://jira.whamcloud.com/browse/LU-10100" title="sanity test_27a: setstripe failed with &quot;error on ioctl 0x8008669a for &#39;*&#39; (3): Invalid argument&quot;" class="issue-link" data-issue-key="LU-10100"><del>LU-10100</del></a>, many of the tests started passing, and test_16a, test_16b, test_17 began crashing, but only because they actually started doing something rather than exiting as soon as the "<tt>lfs setstripe</tt>" command failed. That means reverting <a href="https://jira.whamcloud.com/browse/LU-10100" title="sanity test_27a: setstripe failed with &quot;error on ioctl 0x8008669a for &#39;*&#39; (3): Invalid argument&quot;" class="issue-link" data-issue-key="LU-10100"><del>LU-10100</del></a> would not <em>improve</em> the situation, as much as revert to a state where 100% of the tests are broken on ppc64 compared to only the PFL tests.</p> <p>I'm going to push a debug patch to see if it can identify where the missing swabbing is, which will hopefully allow us to solve the problem. The current issue is that the machine state is so fatally corrupted that the machine reboots without actually printing any stack traces of what is being run.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37494" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37494</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13205" title="sanity-pfl test 16a fails with “setstripe /mnt/lustre/d16.sanity-pfl/f16.sanity-pfl.copy failed“" class="issue-link" data-issue-key="LU-13205">LU-13205</a> lov: debug lov buffer swabbing crash<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e17fe2322008ce7520f4496932ea8638018a4261</p> <p>I've captured a stack trace from the crash. It looks like it is calling <tt>DEBUG_REQ()</tt> in <tt>mdc_close()</tt> with a bad message buffer:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 1188.641162] Lustre: DEBUG MARKER: == sanity-pfl test 16b: Verify setstripe/getstripe with YAML config file + overstriping == 23:01:06 (1581548466) [ 1189.084215] LustreError: 3150:0:(pack_generic.c:2488:lustre_swab_lov_comp_md_v1()) lov: invalid magic 0x1 for component 3 [ 1189.087595] LustreError: 3150:0:(pack_generic.c:1199:lustre_msg_get_transno()) incorrect message magic: 00000200 [ 1189.087680] LustreError: 3150:0:(pack_generic.c:1089:lustre_msg_get_opc()) incorrect message magic: 00000200 (msg:c0000000049cb800) [ 1189.087807] ------------[ cut here ]------------ [ 1189.087904] WARNING: CPU: 1 PID: 3150 at lustre/ptlrpc/pack_generic.c:1090 .lustre_msg_get_opc+0x108/0x330 [ptlrpc] CPU: 1 PID: 3150 Comm: lfs Kdump: loaded Tainted: G OE ------------ 3.10.0-1062.9.1.el7.ppc64 #1 Call Trace: .lustre_msg_get_opc+0xf8/0x330 [ptlrpc] (unreliable) ._debug_req+0x674/0x830 [ptlrpc] .mdc_close+0x568/0x1150 [mdc] .lmv_close+0x238/0x580 [lmv] .ll_close_inode_openhandle+0xb7c/0x1380 [lustre] .ll_md_real_close+0x190/0x330 [lustre] .ll_file_release+0x728/0xcf0 [lustre] .ll_dir_release+0x15c/0x210 [lustre] .__fput+0xe8/0x330 </pre> </div></div> Duplicate LU-13207 Related LU-10100 LU-13215 LU-13186 Development Rank 1|i00t8v: Rank (Obsolete) 9223372036854775807 Severity