https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11933 Lustre <p>This issue was created by maloo for Alexander Boyko &lt;c17825@cray.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/3f081d66-2971-11e9-a318-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3f081d66-2971-11e9-a318-52540065bddc</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 6056.684483] Lustre: DEBUG MARKER: == sanity-hsm test 406: attempting to migrate HSM archived files is safe ============================= 14:13:40 (1549376020) [ 6060.313122] LustreError: 4422:0:(osd_index.c:1117:osd_declare_dir_delete()) ASSERTION( dt_object_exists(dt) ) failed: [ 6060.315139] LustreError: 4422:0:(osd_index.c:1117:osd_declare_dir_delete()) LBUG [ 6060.316456] Pid: 4422, comm: mdt00_000 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Jan 11 05:58:47 UTC 2019 [ 6060.318137] Call Trace: [ 6060.318641] [&lt;ffffffffc0a127cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [ 6060.319891] [&lt;ffffffffc0a1287c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [ 6060.321802] [&lt;ffffffffc1012acd&gt;] osd_declare_dir_delete+0x27d/0x2b0 [osd_zfs] [ 6060.323583] [&lt;ffffffffc129870e&gt;] lod_sub_declare_delete+0xbe/0x1f0 [lod] [ 6060.324964] [&lt;ffffffffc1273be4&gt;] lod_declare_delete+0x24/0x30 [lod] [ 6060.326378] [&lt;ffffffffc130964f&gt;] mdd_orphan_declare_insert+0x33f/0x690 [mdd] [ 6060.327797] [&lt;ffffffffc12f0fe8&gt;] mdd_declare_create+0xa48/0xcb0 [mdd] [ 6060.329315] [&lt;ffffffffc1310d8d&gt;] mdd_declare_migrate_create+0x159/0x3ad [mdd] [ 6060.330679] [&lt;ffffffffc12f61bb&gt;] mdd_migrate+0xe9b/0x1810 [mdd] [ 6060.332092] [&lt;ffffffffc1187786&gt;] mdt_reint_migrate_internal+0xf56/0x14e0 [mdt] [ 6060.333573] [&lt;ffffffffc1187f75&gt;] mdt_reint_rename_or_migrate.isra.51+0x265/0x860 [mdt] [ 6060.335338] [&lt;ffffffffc1188580&gt;] mdt_reint_migrate+0x10/0x20 [mdt] [ 6060.336542] [&lt;ffffffffc1188633&gt;] mdt_reint_rec+0x83/0x210 [mdt] [ 6060.337928] [&lt;ffffffffc1165183&gt;] mdt_reint_internal+0x6e3/0xaf0 [mdt] [ 6060.339487] [&lt;ffffffffc11704e7&gt;] mdt_reint+0x67/0x140 [mdt] [ 6060.340551] [&lt;ffffffffc0e1740a&gt;] tgt_request_handle+0xafa/0x1590 [ptlrpc] [ 6060.342292] [&lt;ffffffffc0dba99e&gt;] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] [ 6060.343705] [&lt;ffffffffc0dbe47c&gt;] ptlrpc_main+0xbbc/0x2090 [ptlrpc] [ 6060.344944] [&lt;ffffffffa52c1c31&gt;] kthread+0xd1/0xe0 [ 6060.345964] [&lt;ffffffffa5974c37&gt;] ret_from_fork_nospec_end+0x0/0x39 [ 6060.347201] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_230b - crashed during sanity test_230b<br/> sanity-hsm test_406 - crashed during sanity-hsm test_406</p> LU-11933

sanity-hsm: test_406 sanity: test_230b osd_declare_dir_delete() ASSERTION( dt_object_exists(dt) )

Bug Major Resolved Fixed Lai Siyao Alexander Boyko Wed, 6 Feb 2019 07:20:16 +0000 Wed, 15 Jan 2020 13:58:49 +0000 Sat, 28 Sep 2019 03:33:40 +0000 Lustre 2.13.0 Lustre 2.12.3 Lustre 2.13.0 Lustre 2.12.4 0 15 <p>Another occurence:<br/> <a href="https://testing.whamcloud.com/test_sets/55bfe258-4be2-11e9-9646-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/55bfe258-4be2-11e9-9646-52540065bddc</a></p> <p><a href="https://testing.whamcloud.com/test_sessions/d5ae56c7-e1fa-4073-9937-dec69a43fb8f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/d5ae56c7-e1fa-4073-9937-dec69a43fb8f</a></p> <p><a href="https://testing.whamcloud.com/test_sets/e307dc3a-44ec-11e9-a256-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e307dc3a-44ec-11e9-a256-52540065bddc</a></p> <p>Just got one more, <a href="https://testing.whamcloud.com/test_sets/09b1fc0a-5b9d-11e9-9720-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/09b1fc0a-5b9d-11e9-9720-52540065bddc</a> .</p> <p>sanity test 230b hit the same assertion on master branch:<br/> <a href="https://testing.whamcloud.com/test_sets/98c37742-653a-11e9-a6f2-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/98c37742-653a-11e9-a6f2-52540065bddc</a></p> <p>this issue is still causing these kind of crashes.</p> <p>+1 on master branch: <a href="https://testing.whamcloud.com/test_sets/e02cf1b6-b603-11e9-b753-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e02cf1b6-b603-11e9-b753-52540065bddc</a></p> <p><a href="https://testing.whamcloud.com/test_sets/81a7f204-b9a9-11e9-a2b6-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/81a7f204-b9a9-11e9-a2b6-52540065bddc</a></p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=laisiyao" class="user-hover" rel="laisiyao">laisiyao</a>,</p> <p>Do you think this crash is related to <a href="https://review.whamcloud.com/#/c/35360/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/35360/</a> / <a href="https://jira.whamcloud.com/browse/LU-12485" title="(osd_handler.c:2146:osd_object_release()) ASSERTION( !(o-&gt;oo_destroyed == 0 &amp;&amp; o-&gt;oo_inode &amp;&amp; o-&gt;oo_inode-&gt;i_nlink == 0) ) faile" class="issue-link" data-issue-key="LU-12485"><del>LU-12485</del></a>?  It's ... similar looking, at least.</p> <p>No, it's related with volatile file migration, I'll check details later.</p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/149b75f8-c5a4-11e9-a2b6-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/149b75f8-c5a4-11e9-a2b6-52540065bddc</a></p> <p>+1 on b2_12 <a href="https://testing.whamcloud.com/test_sessions/26c1e599-2cb2-4191-9492-37ca65e906d9" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/26c1e599-2cb2-4191-9492-37ca65e906d9</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/3c2314e6-ca9a-11e9-90ad-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3c2314e6-ca9a-11e9-90ad-52540065bddc</a> sanity test_230b</p> <p>It looks like this is only crashing during directory migration, mostly in sanity test_230b and much less often in sanity-hsm test_406:<br/> <a href="https://testing.whamcloud.com/sub_tests/query?utf8=%E2%9C%93&amp;warn%5Bnotice%5D=&amp;test_set_script_id=&amp;status%5B%5D=CRASH&amp;query_bugs=LU-11933&amp;horizon=2332800&amp;commit=Update+results&amp;buggable_class=SubTest&amp;num_results=250" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/sub_tests/query?utf8=%E2%9C%93&amp;warn%5Bnotice%5D=&amp;test_set_script_id=&amp;status%5B%5D=CRASH&amp;query_bugs=LU-11933&amp;horizon=2332800&amp;commit=Update+results&amp;buggable_class=SubTest&amp;num_results=250</a></p> <p>It's hit about once a day - not terrible, but probably deserves some attention.</p> <p>Lai Siyao (lai.siyao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36154" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36154</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11933" title="sanity-hsm: test_406 sanity: test_230b osd_declare_dir_delete() ASSERTION( dt_object_exists(dt) )" class="issue-link" data-issue-key="LU-11933"><del>LU-11933</del></a> mdt: clear sp_cr_flags in migrate unpack<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9f8a7b36fd9aeca8e8a752d92091da4de988db0f</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36154/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36154/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11933" title="sanity-hsm: test_406 sanity: test_230b osd_declare_dir_delete() ASSERTION( dt_object_exists(dt) )" class="issue-link" data-issue-key="LU-11933"><del>LU-11933</del></a> mdt: clear sp_cr_flags in migrate unpack<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d4da3b55a8303d937828e74341b3ab5c4dfd52b2</p> <p>Landed for 2.13</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/36398" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36398</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11933" title="sanity-hsm: test_406 sanity: test_230b osd_declare_dir_delete() ASSERTION( dt_object_exists(dt) )" class="issue-link" data-issue-key="LU-11933"><del>LU-11933</del></a> mdt: clear sp_cr_flags in migrate unpack<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 89268c751691065c23ca9a05726f08618c1bbd7c</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36399" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36399</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11933" title="sanity-hsm: test_406 sanity: test_230b osd_declare_dir_delete() ASSERTION( dt_object_exists(dt) )" class="issue-link" data-issue-key="LU-11933"><del>LU-11933</del></a> mdt: clear sp_cr_flags in migrate unpack<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 3bf95822087874ed29f9e770f683183e2b11f750</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36399/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36399/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11933" title="sanity-hsm: test_406 sanity: test_230b osd_declare_dir_delete() ASSERTION( dt_object_exists(dt) )" class="issue-link" data-issue-key="LU-11933"><del>LU-11933</del></a> mdt: clear sp_cr_flags in migrate unpack<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: df9f55d02237821ecfa554eb257b8efc0a84f77d</p> Related LU-12617 Development Rank 1|i00b33: Rank (Obsolete) 9223372036854775807 Severity