https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10134 Lustre <p>Soak mds_failover test run, soak-9 powered off, MDT0001 failed over to soak-8</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">2017-10-17 22:18:47,423:fsmgmt.fsmgmt:INFO Mounting soaked-MDT0001 on soak-8 Soak-8 log: Oct 17 22:19:43 soak-8 kernel: Lustre: soaked-MDT0001: Will be in recovery <span class="code-keyword">for</span> at least 2:30, or until 36 clients reconnect Oct 17 22:19:51 soak-8 kernel: Lustre: soaked-MDT0001: Connection restored to 7553a398-a79b-c096-e685-e7284e2b17df (at 172.16.1.47@o2ib1) </pre> </div></div> <p>Recovery completes, system hits LBUG and dies:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Oct 17 22:21:03 soak-8 kernel: Lustre: soaked-MDT0001: Recovery over after 1:20, of 36 clients 36 recovered and 0 were evicted. Oct 17 22:21:08 soak-8 kernel: LustreError: 3745:0:(lfsck_namespace.c:4571:lfsck_namespace_double_scan()) ASSERTION( list_empty(&amp;lad-&gt;lad_req_list) ) failed: Oct 17 22:21:08 soak-8 kernel: LustreError: 3745:0:(lfsck_namespace.c:4571:lfsck_namespace_double_scan()) LBUG Oct 17 22:21:08 soak-8 kernel: Pid: 3745, comm: lfsck Oct 17 22:21:08 soak-8 kernel: #012Call Trace: Oct 17 22:21:08 soak-8 kernel: [&lt;ffffffffc0dc37ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] Oct 17 22:21:08 soak-8 kernel: [&lt;ffffffffc0dc383c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] Oct 17 22:21:08 soak-8 kernel: [&lt;ffffffffc14ef398&gt;] lfsck_namespace_double_scan+0x108/0x140 [lfsck] Oct 17 22:21:08 soak-8 kernel: [&lt;ffffffffc14e65a9&gt;] lfsck_double_scan+0x59/0x200 [lfsck] Oct 17 22:21:08 soak-8 kernel: [&lt;ffffffffc143550a&gt;] ? osd_otable_it_fini+0xca/0x240 [osd_ldiskfs] Oct 17 22:21:09 soak-8 kernel: [&lt;ffffffff811deec3&gt;] ? kfree+0x103/0x140 Oct 17 22:21:09 soak-8 kernel: [&lt;ffffffffc14eb134&gt;] lfsck_master_engine+0x494/0x12b0 [lfsck] Oct 17 22:21:09 soak-8 kernel: [&lt;ffffffff810c4810&gt;] ? default_wake_function+0x0/0x20 Oct 17 22:21:09 soak-8 kernel: [&lt;ffffffffc14eaca0&gt;] ? lfsck_master_engine+0x0/0x12b0 [lfsck] Oct 17 22:21:09 soak-8 kernel: [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 Oct 17 22:21:09 soak-8 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 Oct 17 22:21:09 soak-8 kernel: [&lt;ffffffff816b4f18&gt;] ret_from_fork+0x58/0x90 Oct 17 22:21:09 soak-8 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 Oct 17 22:21:09 soak-8 kernel: Oct 17 22:21:09 soak-8 kernel: Kernel panic - not syncing: LBUG </pre> </div></div> <p>Crash dump from soak-8 is available on spirit cluster at: /scratch/dumps/soak-8.spirit.hpdd.intel.com/10.10.1.108-2017-10-17-22:21:34<br/> vmcore-dmesg.txt attached</p> Soak cluster, lustre-master build 3654 LU-10134

LBUG lfsck_namespace_double_scan()) ASSERTION( list_empty(&lad->lad_req_list) ) failed:

Bug Critical Resolved Fixed nasf Cliff White soak Wed, 18 Oct 2017 00:06:52 +0000 Thu, 4 Jan 2018 18:28:39 +0000 Sun, 17 Dec 2017 15:57:20 +0000 Lustre 2.11.0 Lustre 2.10.2 Lustre 2.11.0 Lustre 2.10.3 0 5 <p>Fan Yong - Would you please comment on this issue?</p> <p>Thank you</p> <p>Hit this issue again with latest 2.8-fe</p> <p>Hit this issue again with 2.10.55. Seems to be pretty constant<br/> vmcore is available on soak.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Nov 15 15:20:09 soak-9 systemd: Stopping User Slice of root. Nov 15 15:20:18 soak-9 kernel: Lustre: soaked-MDT0000: Recovery over after 1:42, of 32 clients 32 recovered and 0 were evicted. Nov 15 15:20:18 soak-9 kernel: LustreError: 27404:0:(lfsck_namespace.c:4571:lfsck_namespace_double_scan()) ASSERTION( list_empty(&amp;lad-&gt;lad_req_list) ) failed: Nov 15 15:20:18 soak-9 kernel: LustreError: 27404:0:(lfsck_namespace.c:4571:lfsck_namespace_double_scan()) LBUG Nov 15 15:20:18 soak-9 kernel: Pid: 27404, comm: lfsck Nov 15 15:20:18 soak-9 kernel: #012Call Trace: Nov 15 15:20:18 soak-9 kernel: [&lt;ffffffffc0dcd7ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] Nov 15 15:20:18 soak-9 kernel: [&lt;ffffffffc0dcd83c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] Nov 15 15:20:18 soak-9 kernel: [&lt;ffffffffc14a0398&gt;] lfsck_namespace_double_scan+0x108/0x140 [lfsck] Nov 15 15:20:18 soak-9 kernel: [&lt;ffffffffc14975a9&gt;] lfsck_double_scan+0x59/0x200 [lfsck] Nov 15 15:20:18 soak-9 kernel: [&lt;ffffffffc143c4ea&gt;] ? osd_otable_it_fini+0xca/0x240 [osd_ldiskfs] Nov 15 15:20:18 soak-9 kernel: [&lt;ffffffff811dee33&gt;] ? kfree+0x103/0x140 Nov 15 15:20:18 soak-9 kernel: [&lt;ffffffffc149c134&gt;] lfsck_master_engine+0x494/0x12b0 [lfsck] Nov 15 15:20:18 soak-9 kernel: [&lt;ffffffff810c4820&gt;] ? default_wake_function+0x0/0x20 Nov 15 15:20:18 soak-9 kernel: [&lt;ffffffffc149bca0&gt;] ? lfsck_master_engine+0x0/0x12b0 [lfsck] Nov 15 15:20:18 soak-9 kernel: [&lt;ffffffff810b099f&gt;] kthread+0xcf/0xe0 Nov 15 15:20:18 soak-9 kernel: [&lt;ffffffff810b08d0&gt;] ? kthread+0x0/0xe0 Nov 15 15:20:18 soak-9 kernel: [&lt;ffffffff816b4fd8&gt;] ret_from_fork+0x58/0x90 Nov 15 15:20:18 soak-9 kernel: [&lt;ffffffff810b08d0&gt;] ? kthread+0x0/0xe0 Nov 15 15:20:18 soak-9 kernel: Nov 15 15:20:18 soak-9 kernel: Kernel panic - not syncing: LBUG </pre> </div></div> <p>After this incident, the system is completely non-recoverable. Any attempt to re-mount MDT0 causes this LBUG to hit again, and again. Multiple core dumps are available on spirit. </p> <p>Hi Fan Yong,</p> <p>Can you please look at this issue? It has become a roadblock for soak testing on master.</p> <p>Thanks.<br/> Joe</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30165" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30165</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10134" title="LBUG lfsck_namespace_double_scan()) ASSERTION( list_empty(&amp;lad-&gt;lad_req_list) ) failed:" class="issue-link" data-issue-key="LU-10134"><del>LU-10134</del></a> lfsck: not add requests if engine out of work<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bd6411b44714543fae3e841ffd055e67f105d24b</p> <p>We just hit this assert on b2.10.2-RC1, crash dump is available on Spirit</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"></pre> </div></div> <p>[ 1256.526444] Lustre: Failing over soaked-MDT0000<br/> [ 1256.536364] LustreError: 2796:0:(lfsck_namespace.c:4582:lfsck_namespace_double_scan()) ASSERTION( list_empty(&amp;lad-&gt;lad_req_list) ) failed:<br/> [ 1256.537017] LustreError: 2303:0:(ldlm_lockd.c:1415:ldlm_handle_enqueue0()) ### lock on destroyed export ffff88080e0d1800 ns: mdt-soaked-MDT0000_UUID lock: ffff880811089600/0x18106beaab642978 lrc: 3/0,0 mode: PR/PR res: <span class="error">&#91;0x200000405:0x1:0x0&#93;</span>.0x0 bits 0x13 rrc: 2 type: IBT flags: 0x50200000000000 nid: 192.168.1.117@o2ib remote: 0xd36fb2fda127ff7 expref: 3 pid: 2303 timeout: 0 lvb_type: 0<br/> [ 1256.580979] Lustre: soaked-MDT0000: Not available for connect from 192.168.1.117@o2ib (stopping)<br/> [ 1256.614544] LustreError: 2796:0:(lfsck_namespace.c:4582:lfsck_namespace_double_scan()) LBUG<br/> [ 1256.625905] Pid: 2796, comm: lfsck<br/> [ 1256.631630]<br/> Call Trace:<br/> [ 1256.639771] <span class="error">&#91;&lt;ffffffffc0dd57ae&gt;&#93;</span> libcfs_call_trace+0x4e/0x60 <span class="error">&#91;libcfs&#93;</span><br/> [ 1256.648850] <span class="error">&#91;&lt;ffffffffc0dd583c&gt;&#93;</span> lbug_with_loc+0x4c/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> [ 1256.657599] <span class="error">&#91;&lt;ffffffffc14964c8&gt;&#93;</span> lfsck_namespace_double_scan+0x108/0x140 <span class="error">&#91;lfsck&#93;</span><br/> [ 1256.667697] <span class="error">&#91;&lt;ffffffffc148d6a9&gt;&#93;</span> lfsck_double_scan+0x59/0x200 <span class="error">&#91;lfsck&#93;</span><br/> [ 1256.676632] <span class="error">&#91;&lt;ffffffffc143344a&gt;&#93;</span> ? osd_otable_it_fini+0xca/0x240 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> [ 1256.686386] <span class="error">&#91;&lt;ffffffff811dee33&gt;&#93;</span> ? kfree+0x103/0x140<br/> [ 1256.693515] <span class="error">&#91;&lt;ffffffffc1492204&gt;&#93;</span> lfsck_master_engine+0x494/0x12f0 <span class="error">&#91;lfsck&#93;</span><br/> [ 1256.702670] <span class="error">&#91;&lt;ffffffff810c4820&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> [ 1256.710967] <span class="error">&#91;&lt;ffffffffc1491d70&gt;&#93;</span> ? lfsck_master_engine+0x0/0x12f0 <span class="error">&#91;lfsck&#93;</span><br/> [ 1256.720038] <span class="error">&#91;&lt;ffffffff810b099f&gt;&#93;</span> kthread+0xcf/0xe0<br/> [ 1256.726787] <span class="error">&#91;&lt;ffffffff810bf114&gt;&#93;</span> ? finish_task_switch+0x54/0x160<br/> [ 1256.734873] <span class="error">&#91;&lt;ffffffff810b08d0&gt;&#93;</span> ? kthread+0x0/0xe0<br/> [ 1256.741642] <span class="error">&#91;&lt;ffffffff816b4fd8&gt;&#93;</span> ret_from_fork+0x58/0x90<br/> [ 1256.748880] <span class="error">&#91;&lt;ffffffff810b08d0&gt;&#93;</span> ? kthread+0x0/0xe0<br/> [ 1256.755609]<br/> [ 1256.758443] Kernel panic - not syncing: LBUG<br/> {code]</p> <p>After the first assert hits, attempting to restart the systems results in further asserts. Can't start soak at this time, due to this. 3 out of our 4 MDS are now halted.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">soak-10.log:Dec 5 16:44:33 soak-10 kernel: LustreError: 2593:0:(lfsck_namespace.c:4582:lfsck_namespace_double_scan()) LBUG soak-10.log:Dec 5 16:44:34 soak-10 kernel: Kernel panic - not syncing: LBUG soak-8.log:Dec 5 16:44:31 soak-8 kernel: LustreError: 2529:0:(lfsck_namespace.c:4582:lfsck_namespace_double_scan()) LBUG soak-8.log:Dec 5 16:44:32 soak-8 kernel: Kernel panic - not syncing: LBUG soak-9.log:Dec 5 00:32:11 soak-9 kernel: LustreError: 2796:0:(lfsck_namespace.c:4582:lfsck_namespace_double_scan()) LBUG soak-9.log:Dec 5 00:32:11 soak-9 kernel: Kernel panic - not syncing: LBUG </pre> </div></div> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30375" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30375</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10134" title="LBUG lfsck_namespace_double_scan()) ASSERTION( list_empty(&amp;lad-&gt;lad_req_list) ) failed:" class="issue-link" data-issue-key="LU-10134"><del>LU-10134</del></a> lfsck: not add requests if engine out of work<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 90df962c58fb18367291dfff8698fdb1e70d9188</p> <p>I ported Fan Yong's patch to b2_10 to see if this will help with the issues seen on the soak cluster.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30165/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30165/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10134" title="LBUG lfsck_namespace_double_scan()) ASSERTION( list_empty(&amp;lad-&gt;lad_req_list) ) failed:" class="issue-link" data-issue-key="LU-10134"><del>LU-10134</del></a> lfsck: not add requests if engine out of work<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f22a0ab6c37db2d983451ec01e869ed8d3226cb2</p> <p>Landed for 2.11</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/30375/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30375/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10134" title="LBUG lfsck_namespace_double_scan()) ASSERTION( list_empty(&amp;lad-&gt;lad_req_list) ) failed:" class="issue-link" data-issue-key="LU-10134"><del>LU-10134</del></a> lfsck: not add requests if engine out of work<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 02065756c83057d3acc35792b744078f2739025d</p> Related LU-8647 Development Rank 1|hzzm1z: Rank (Obsolete) 9223372036854775807 Severity