https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10437 Lustre <p>session: <a href="https://testing.hpdd.intel.com/test_sessions/cb5f13e0-9177-4f4e-9a26-d197001db0c0" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/cb5f13e0-9177-4f4e-9a26-d197001db0c0</a><br/> test set: <a href="https://testing.hpdd.intel.com/test_sets/5922f1c0-e4ef-11e7-8027-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5922f1c0-e4ef-11e7-8027-52540065bddc</a></p> <p>From test_log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>copying /usr/share/dbench/client.txt to /mnt/lustre/d8.sanity-pfl/client.txt cp: error writing '/mnt/lustre/d8.sanity-pfl/client.txt': Invalid argument cp: failed to extend '/mnt/lustre/d8.sanity-pfl/client.txt': Invalid argument Trace dump: = rundbench:55:main() sanity-pfl: FAIL: test-framework exiting on error sanity-pfl test_8: @@@@@@ FAIL: dbench failed Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:5328:error() = /usr/lib64/lustre/tests/sanity-pfl.sh:333:test_8() = /usr/lib64/lustre/tests/test-framework.sh:5604:run_one() = /usr/lib64/lustre/tests/test-framework.sh:5643:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:5490:run_test() = /usr/lib64/lustre/tests/sanity-pfl.sh:337:main() </pre> </div></div> onyx, full interop<br/> servers: el7.4, ldiskfs, branch b2_10, v2.10.2, b52<br/> clients: el7.4, branch master, v2.10.56, b3678<br/> LU-10437

sanity-pfl test_8: dbench failed

Bug Minor Resolved Fixed WC Triage James Casper Tue, 26 Dec 2017 18:52:01 +0000 Wed, 29 Aug 2018 17:34:01 +0000 Sat, 20 Jan 2018 16:35:43 +0000 Lustre 2.11.0 Lustre 2.11.0 Lustre 2.10.4 0 5 <p>Dmesg log on client node:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 514:0:(lov_object.c:1220:lov_layout_change()) lustre-clilov-ffff88005daa9000: cannot apply new layout on [0x200062e21:0x2735:0x0] : rc = -22 LustreError: 514:0:(vvp_io.c:1495:vvp_io_init()) lustre: refresh file layout [0x200062e21:0x2735:0x0] error -22. </pre> </div></div> <p>Debug log on client node:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000080:00200000:1.0:1513687522.574612:0:514:0:(vvp_io.c:312:vvp_io_fini()) [0x200062e21:0x2735:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0 00020000:00020000:1.0:1513687522.578220:0:514:0:(lov_object.c:1220:lov_layout_change()) lustre-clilov-ffff88005daa9000: cannot apply new layout on [0x200062e21:0x2735:0x0] : rc = -22 00010000:00010000:1.0:1513687522.579782:0:514:0:(ldlm_lock.c:800:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: ?? lock: ffff88005cf42240/0xed5adfd7af8ec0c4 lrc: 3/1,0 mode: CR/CR res: ?? rrc=?? type: ??? flags: 0x10000000000000 nid: local remote: 0x4869a6ac1a9ab3d4 expref: -99 pid: 514 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1513687522.579786:0:514:0:(ldlm_lock.c:873:ldlm_lock_decref_internal()) ### add lock into lru list ns: ?? lock: ffff88005cf42240/0xed5adfd7af8ec0c4 lrc: 2/0,0 mode: CR/CR res: ?? rrc=?? type: ??? flags: 0x10000000000000 nid: local remote: 0x4869a6ac1a9ab3d4 expref: -99 pid: 514 timeout: 0 lvb_type: 3 00000080:00020000:1.0:1513687522.579791:0:514:0:(vvp_io.c:1495:vvp_io_init()) lustre: refresh file layout [0x200062e21:0x2735:0x0] error -22. 00000080:00200000:1.0:1513687522.580937:0:514:0:(vvp_io.c:312:vvp_io_fini()) [0x200062e21:0x2735:0x0] ignore/verify layout 0/0, layout version -2 need write layout 0, restore needed 0 00000080:00200000:1.0:1513687522.580940:0:514:0:(file.c:1423:ll_file_io_generic()) client.txt: 2 io complete with rc: -22, result: 0, restart: 0 00000080:00200000:1.0:1513687522.580942:0:514:0:(file.c:1459:ll_file_io_generic()) client.txt: write *ppos: 16777216, pos: 16777216, ret: 0, rc: -22 </pre> </div></div> <p>Dmesg log on MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 18669:0:(mdt_lvb.c:163:mdt_lvbo_fill()) lustre-MDT0000: expected 368 actual 344. </pre> </div></div> <p>sanity-pfl test 15 in the same interop test session hit the same failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity-pfl test 15: Verify component options for lfs find ========================================= 12:46:26 (1513687586) dd: error writing '/mnt/lustre/d15.sanity-pfl/f1': Invalid argument </pre> </div></div> <p>Debug log on client node:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000080:00200000:1.0:1513687587.114545:0:6996:0:(vvp_io.c:312:vvp_io_fini()) [0x200062e21:0x2743:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0 00020000:00020000:1.0:1513687587.114768:0:6996:0:(lov_object.c:1220:lov_layout_change()) lustre-clilov-ffff88005daa9000: cannot apply new layout on [0x200062e21:0x2743:0x0] : rc = -22 00010000:00010000:1.0:1513687587.116338:0:6996:0:(ldlm_lock.c:800:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: ?? lock: ffff88005cf43440/0xed5adfd7af8ec4ec lrc: 3/1,0 mode: CR/CR res: ?? rrc=?? type: ??? flags: 0x10000000000000 nid: local remote: 0x4869a6ac1a9aca2b expref: -99 pid: 6996 timeout: 0 lvb_type: 3 00010000:00010000:1.0:1513687587.116341:0:6996:0:(ldlm_lock.c:873:ldlm_lock_decref_internal()) ### add lock into lru list ns: ?? lock: ffff88005cf43440/0xed5adfd7af8ec4ec lrc: 2/0,0 mode: CR/CR res: ?? rrc=?? type: ??? flags: 0x10000000000000 nid: local remote: 0x4869a6ac1a9aca2b expref: -99 pid: 6996 timeout: 0 lvb_type: 3 00000080:00020000:1.0:1513687587.116345:0:6996:0:(vvp_io.c:1495:vvp_io_init()) lustre: refresh file layout [0x200062e21:0x2743:0x0] error -22. 00000080:00200000:1.0:1513687587.117486:0:6996:0:(vvp_io.c:312:vvp_io_fini()) [0x200062e21:0x2743:0x0] ignore/verify layout 0/0, layout version -2 need write layout 0, restore needed 0 00000080:00200000:1.0:1513687587.117488:0:6996:0:(file.c:1423:ll_file_io_generic()) f1: 2 io complete with rc: -22, result: 0, restart: 0 00000080:00200000:1.0:1513687587.117489:0:6996:0:(file.c:1459:ll_file_io_generic()) f1: write *ppos: 1048576, pos: 1048576, ret: 0, rc: -22 </pre> </div></div> <p>it turned out that the b2_10 branch doesn't clear either <tt>lcm_flags</tt> or <tt>lcm_padding</tt> fields when packing layout on the server side. The corresponding code is in <tt>lod_generate_lovea()</tt>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lcm = (struct lov_comp_md_v1 *)lmm; lcm-&gt;lcm_magic = cpu_to_le32(LOV_MAGIC_COMP_V1); lcm-&gt;lcm_entry_count = cpu_to_le16(comp_cnt); offset = sizeof(*lcm) + sizeof(*lcme) * comp_cnt; LASSERT(offset % sizeof(__u64) == 0); </pre> </div></div> <p>This will confuse b2_11 clients because <tt>lcm_flags</tt> and <tt>lcm_mirror_count</tt> would be random numbers and then will not pass sanity check.</p> <p>I think the best way to fix this problem is to create a patch to clear the corresponding fields in b2_10.</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30784" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30784</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10437" title="sanity-pfl test_8: dbench failed" class="issue-link" data-issue-key="LU-10437"><del>LU-10437</del></a> lod: clear layout header when generating layout<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 2eb12e91fea8f57b1c00dac7d756bedcda4aee1f</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30785" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30785</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10437" title="sanity-pfl test_8: dbench failed" class="issue-link" data-issue-key="LU-10437"><del>LU-10437</del></a> lod: clear layout header when generating layout<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b34f3ccad87cbcdd9b0c2bd4a84d6221735dc9dd</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30785/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30785/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10437" title="sanity-pfl test_8: dbench failed" class="issue-link" data-issue-key="LU-10437"><del>LU-10437</del></a> lod: clear layout header when generating layout<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 47d6ce20cfd8b04f20f7fc7accc39b3902780900</p> <p>Landed for 2.11</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/30784/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30784/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10437" title="sanity-pfl test_8: dbench failed" class="issue-link" data-issue-key="LU-10437"><del>LU-10437</del></a> lod: clear layout header when generating layout<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 0e38e97e2c4209ac31f3f6f9bc245da9a991006c</p> Duplicate LU-10439 Related LU-11291 Development Rank 1|hzzq5j: Rank (Obsolete) 9223372036854775807 Severity