https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10143 Lustre <p><a href="https://testing.hpdd.intel.com/test_sessions/2f6027c4-8e9a-4949-af3d-2f8d04940c9c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/2f6027c4-8e9a-4949-af3d-2f8d04940c9c</a></p> <p>replay-single, test_80d: Timeout occurred after 155 mins, last suite running was replay-single, restarting cluster to continue tests LBUG</p> <p>From mds console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 5748.826345] LustreError: 28616:0:(dt_object.h:2166:dt_declare_record_write()) ASSERTION( dt-&gt;do_body_ops ) failed: [ 5748.828884] LustreError: 28616:0:(dt_object.h:2166:dt_declare_record_write()) LBUG [ 5748.831194] Pid: 28616, comm: mdt_out00_003 [ 5748.833233] [ 5748.833233] Call Trace: [ 5748.836889] [&lt;ffffffffc06917ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] [ 5748.839015] [&lt;ffffffffc069183c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] [ 5748.841118] [&lt;ffffffffc0ef1743&gt;] out_write_add_exec+0x133/0x1b0 [ptlrpc] [ 5748.843217] [&lt;ffffffffc0ee84a3&gt;] out_write+0x333/0x370 [ptlrpc] [ 5748.845224] [&lt;ffffffffc0eeb1c4&gt;] out_handle+0x1304/0x1920 [ptlrpc] [ 5748.847209] [&lt;ffffffffc0e7d4a2&gt;] ? lustre_msg_get_opc+0x22/0xf0 [ptlrpc] [ 5748.849234] [&lt;ffffffffc0ee0d49&gt;] ? tgt_request_preprocess.isra.26+0x299/0x7a0 [ptlrpc] [ 5748.851305] [&lt;ffffffffc0ee2475&gt;] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 5748.853263] [&lt;ffffffffc0e8b37e&gt;] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] [ 5748.855236] [&lt;ffffffff810ba588&gt;] ? __wake_up_common+0x58/0x90 [ 5748.857056] [&lt;ffffffffc0e8eb22&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 5748.858924] [&lt;ffffffffc0e8e090&gt;] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 5748.860737] [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 [ 5748.862471] [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 [ 5748.864195] [&lt;ffffffff816b4f18&gt;] ret_from_fork+0x58/0x90 [ 5748.865926] [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 [ 5748.867590] [ 5748.868971] Kernel panic - not syncing: LBUG [ 5748.869962] CPU: 1 PID: 28616 Comm: mdt_out00_003 Tainted: P OE ------------ 3.10.0-693.2.2.el7_lustre.x86_64 #1 [ 5748.869962] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 [ 5748.869962] ffff88007c7daf00 000000006e5a06dd ffff88005feefa88 ffffffff816a3d8d [ 5748.869962] ffff88005feefb08 ffffffff8169dc74 ffffffff00000008 ffff88005feefb18 [ 5748.869962] ffff88005feefab8 000000006e5a06dd 000000006e5a06dd ffff88007fd0f8b8 [ 5748.869962] Call Trace: [ 5748.869962] [&lt;ffffffff816a3d8d&gt;] dump_stack+0x19/0x1b [ 5748.869962] [&lt;ffffffff8169dc74&gt;] panic+0xe8/0x20d [ 5748.869962] [&lt;ffffffffc0691854&gt;] lbug_with_loc+0x64/0xb0 [libcfs] [ 5748.869962] [&lt;ffffffffc0ef1743&gt;] out_write_add_exec+0x133/0x1b0 [ptlrpc] [ 5748.869962] [&lt;ffffffffc0ee84a3&gt;] out_write+0x333/0x370 [ptlrpc] [ 5748.869962] [&lt;ffffffffc0eeb1c4&gt;] out_handle+0x1304/0x1920 [ptlrpc] [ 5748.869962] [&lt;ffffffffc0e7d4a2&gt;] ? lustre_msg_get_opc+0x22/0xf0 [ptlrpc] [ 5748.869962] [&lt;ffffffffc0ee0d49&gt;] ? tgt_request_preprocess.isra.26+0x299/0x7a0 [ptlrpc] [ 5748.893323] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 [ 5748.893798] [&lt;ffffffffc0ee2475&gt;] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 5748.893798] [&lt;ffffffffc0e8b37e&gt;] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] [ 5748.893798] [&lt;ffffffff810ba588&gt;] ? __wake_up_common+0x58/0x90 [ 5748.893798] [&lt;ffffffffc0e8eb22&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 5748.893798] [&lt;ffffffffc0e8e090&gt;] ? ptlrpc_register_service+0xe80/0xe80 [ptlrpc] [ 5748.893798] [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 [ 5748.893798] [&lt;ffffffff810b08c0&gt;] ? insert_kthread_work+0x40/0x40 [ 5748.893798] [&lt;ffffffff816b4f18&gt;] ret_from_fork+0x58/0x90 [ 5748.893798] [&lt;ffffffff810b08c0&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> trevis, full DNE<br/> servers: CentOS7.4, zfs, branch master, v2.10.54, b3652<br/> clients: CentOS7.4, branch master, v2.10.54, b3652<br/> LU-10143

LBUG dt_object.h:2166:dt_declare_record_write

Bug Minor Resolved Duplicate Alex Zhuravlev James Casper Thu, 19 Oct 2017 20:09:22 +0000 Fri, 7 Feb 2020 22:40:17 +0000 Fri, 7 Feb 2020 22:40:17 +0000 Lustre 2.11.0 Lustre 2.12.0 Lustre 2.10.5 Lustre 2.10.6 Lustre 2.13.0 Lustre 2.10.7 Lustre 2.12.1 0 13 <p>Hi Alex,</p> <p>Can you please look into this?</p> <p>Thanks.<br/> Joe</p> <p>another instance on master tag-2.11.53 ZFS DNE</p> <p><a href="https://testing.whamcloud.com/test_sets/454a32d6-9097-11e8-a9f7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/454a32d6-9097-11e8-a9f7-52540065bddc</a></p> <p>trying to reproduce locally...</p> <p> </p> <p>+1 on b2_10:<br/> <a href="https://testing.whamcloud.com/test_sets/509ffa68-9d11-11e8-a9f7-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/509ffa68-9d11-11e8-a9f7-52540065bddc</a></p> <p>We have replay-single test_80g crash wth the same stack trace. Logs are at <a href="https://testing.whamcloud.com/test_sets/9c64d894-fdc2-11e8-b837-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/9c64d894-fdc2-11e8-b837-52540065bddc</a></p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33849" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33849</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a> obdclass: additional debug<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2adb23e506c7fd52a1d1255b214e6c3626af1d7c</p> <p>the interesting difference in the code is that osd-ldisfsk sets .do_body_ops unconditionally, right at object initialisation (even if it doesn't exist yet) while osd-zfs sets .do_body_ops only if object exists or declared to be created. but there is one case when object is being destroyed by ZFS and in this case .do_body_ops is not set. with the patch above I'm going to catch this case if the theory is correct.</p> <p>though the next step is not obvious as object (likely llog) is expected to exist at this point.</p> <p> </p> <p>Another one in master:<br/> <a href="https://testing.whamcloud.com/test_sets/9838396a-18e7-11e9-8388-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/9838396a-18e7-11e9-8388-52540065bddc</a></p> <p>learnt how to reproduce locally..</p> <p>so far, tracked this down to FID (SEQ) duplication only with ZFS..</p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34069" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34069</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a> osd-zfs: allocate sequence in advance<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 902fbc22f0b9f3dc17ab48912cb361cd93a40db1</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34069/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34069/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a> osd-zfs: allocate sequence in advance<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 51c449b73994f2bba98ee27ac77f90c9aa846e88</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34267" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34267</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a> osd-zfs: allocate sequence in advance<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 4dd7b58d7cab32364c75df4cafd6d6995846b215</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34285" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34285</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a> tests: Add version check for interop<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a5fac453acc8ddb0a59c4f76341ad7825ecbfd34</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34267/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34267/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a> osd-zfs: allocate sequence in advance<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 98ec90dd316054904f054e3cf88ae0ba8e54a2dd</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34295" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34295</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a> osd-zfs: allocate sequence in advance<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: a1126a4baebf2aa8fcf43c270398a380d3c0b503</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34285/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34285/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a> tests: Add version check for interop<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1e63c2f85da17947d78d4b2fb79cab2bd04b2ca5</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34295/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34295/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a> osd-zfs: allocate sequence in advance<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 6c69952f0654028d129bd76a7a24007ead26610f</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34562" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34562</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a> tests: Add version check for interop<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: fdada0bd114f4355ee79dfe250cc448d34b98ed6</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34562/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34562/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10143" title="LBUG dt_object.h:2166:dt_declare_record_write" class="issue-link" data-issue-key="LU-10143"><del>LU-10143</del></a> tests: Add version check for interop<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: c44b6a5f36c66b3c02846c4ccaa6eca5bf061434</p> <p>+1 with recent master at <a href="https://testing.whamcloud.com/test_sessions/97ef006b-0a64-4710-b57c-83f6318cb9ec" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/97ef006b-0a64-4710-b57c-83f6318cb9ec</a> .</p> <p>+1 on master:<br/> <a href="https://testing.whamcloud.com/test_sets/85fcc268-0728-11ea-8e77-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/85fcc268-0728-11ea-8e77-52540065bddc</a></p> <p>Reopening this ticket because it looks like we are seeing this issue again on master (2.14), IN this case <a href="https://testing.whamcloud.com/test_sets/278f669c-05aa-11ea-bbc3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/278f669c-05aa-11ea-bbc3-52540065bddc</a>, we are seeing replay-single test_118 crash with this LBUG.</p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/1d5271ac-0b5d-11ea-8e77-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1d5271ac-0b5d-11ea-8e77-52540065bddc</a></p> <p>+5 on master in the past week. This seems very likely related to <a href="https://jira.whamcloud.com/browse/LU-9924" title="LBUG dt_object.c:513:dt_record_write" class="issue-link" data-issue-key="LU-9924"><del>LU-9924</del></a>.<br/> <a href="https://testing.whamcloud.com/test_sets/7992c6c2-0cae-11ea-98f1-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/7992c6c2-0cae-11ea-98f1-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/ab5220e6-0cad-11ea-bbc3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/ab5220e6-0cad-11ea-bbc3-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/576bd82e-108f-11ea-8e77-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/576bd82e-108f-11ea-8e77-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/7425bbb6-10a2-11ea-9487-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/7425bbb6-10a2-11ea-9487-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/78f6a1f8-10ae-11ea-98f1-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/78f6a1f8-10ae-11ea-98f1-52540065bddc</a></p> <p>+1 on master replay-single test_118 <a href="https://testing.whamcloud.com/test_sets/ffc091c4-3892-11ea-b1e8-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/ffc091c4-3892-11ea-b1e8-52540065bddc</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/22e755f0-3b97-11ea-80b4-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/22e755f0-3b97-11ea-80b4-52540065bddc</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/8ee4dcc8-415a-11ea-9847-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8ee4dcc8-415a-11ea-9847-52540065bddc</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/9588453c-41a6-11ea-af6a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/9588453c-41a6-11ea-af6a-52540065bddc</a></p> <p>I'm going to close this ticket, since it had patches landed and backported to other branches already. <a href="https://jira.whamcloud.com/browse/LU-13195" title="replay-single test_118: dt_declare_record_write() ASSERTION( dt-&gt;do_body_ops ) failed" class="issue-link" data-issue-key="LU-13195"><del>LU-13195</del></a> can be used for investigating/fixing this (potentially the same) issue on master for 2.14.</p> Duplicate LU-9157 LU-10740 LU-11538 LU-13195 Related LU-11366 LU-7298 LU-9924 Development Rank 1|hzzm7z: Rank (Obsolete) 9223372036854775807 Severity