https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11593 Lustre <p>Hi</p> <p>This is Manish from nasa and we were getting some hangs on ls command and because of that we started running the "lfsck" command on all OSS nodes to clear up stale entries and that crashed all the servers after a long run of lfsck command.</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> PID: 25536  TASK: ffff881bc2abaf70  CPU: 9   COMMAND: <span class="code-quote">"lfsck"</span>  #0 [ffff88151015fbc8] machine_kexec at ffffffff8105b64b  #1 [ffff88151015fc28] __crash_kexec at ffffffff81105342  #2 [ffff88151015fcf8] panic at ffffffff81689aad  #3 [ffff88151015fd78] lbug_with_loc at ffffffffa08938cb [libcfs]  #4 [ffff88151015fd98] lfsck_layout_slave_prep at ffffffffa0cfa6fd [lfsck]  #5 [ffff88151015fdf0] lfsck_master_engine at ffffffffa0cd4624 [lfsck]  #6 [ffff88151015fec8] kthread at ffffffff810b1131  #7 [ffff88151015ff50] ret_from_fork at ffffffff816a14f7</pre> </div></div> <p> </p> <p>I will upload the complete crash dump to ftp site soon.</p> <p>Thank You,</p> <p>                   Manish</p> CentOS Linux release 7.5.1804 (Core) <br/> Kernel Version 3.10.0-693.21.1.el7.20180508.x86_64.lustre2105 <br/> e2fsprogs-libs-1.44.3.wc1-0.el7.x86_64 <br/> e2fsprogs-1.44.3.wc1-0.el7.x86_64 <br/> e2fsprogs-static-1.44.3.wc1-0.el7.x86_64 <br/> e2fsprogs-devel-1.44.3.wc1-0.el7.x86_64 <br/> e2fsprogs-debuginfo-1.44.3.wc1-0.el7.x86_64 LU-11593

LFSCK crashed all OSS servers

Bug Major In Progress Unresolved Lai Siyao Manish Thu, 1 Nov 2018 16:39:08 +0000 Mon, 14 Jun 2021 21:23:49 +0000 Lustre 2.10.5 0 8 <p>Lai</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>The <tt>lfsck_layout_slave_prep()</tt> function has two different <tt>LASSERT()</tt> checks in it, but it isn't clear from the above stack trace which one was triggered. Are there some lines on the console before the stack trace that show what the actual problem was? Based on the <tt>lfsck_layout_slave_prep()</tt> function this was in, it looks like it was at the start of the MDS-&gt;OSS <tt>layout</tt> scanning phase, so it may have already repaired the corrupted LMA structures on the MDS.</p> <p>If the previous LFSCK run has repaired most of the issues with the files, it is not strictly necessary to continue with LFSCK prior to returning the filesystem to service. You might consider to run a "find -uid 0" (or similar, maybe in parallel across each top-level directory from a separate client) on the filesystem to ensure that there are no files that cause the MDS or client to crash when accessed, but cleaning up orphan OST objects is probably a secondary concern at this point. Lustre clients can handle corrupt/unknown file layouts and missing OST objects fairly well (returning an error when such files are accessed).</p> <p>Here is the stack trace from vmcore-dmesg what I see from one of the node.</p> <p>  </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [216597.870804] LustreError: 29043:0:(lfsck_layout.c:5144:lfsck_layout_slave_prep()) ASSERTION( !llsd-&gt;llsd_rbtree_valid ) failed: [216597.870893] LustreError: 29045:0:(lfsck_layout.c:5144:lfsck_layout_slave_prep()) ASSERTION( !llsd-&gt;llsd_rbtree_valid ) failed: [216597.870896] LustreError: 29045:0:(lfsck_layout.c:5144:lfsck_layout_slave_prep()) LBUG [216597.870897] Pid: 29045, comm: lfsck 3.10.0-693.21.1.el7.20180508.x86_64.lustre2105 #1 SMP Mon Aug 27 23:04:41 UTC 2018 [216597.870898] Call Trace: [216597.870916]  [&lt;ffffffff8103a1f2&gt;] save_stack_trace_tsk+0x22/0x40 [216597.870926]  [&lt;ffffffffa086d7cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [216597.870931]  [&lt;ffffffffa086d87c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [216597.870954]  [&lt;ffffffffa10e06fd&gt;] lfsck_layout_slave_prep+0x51d/0x590 [lfsck] [216597.870961]  [&lt;ffffffffa10ba624&gt;] lfsck_master_engine+0x184/0x1360 [lfsck] [216597.870965]  [&lt;ffffffff810b1131&gt;] kthread+0xd1/0xe0 [216597.870968]  [&lt;ffffffff816a14f7&gt;] ret_from_fork+0x77/0xb0 [216597.870989]  [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [216597.870990] Kernel panic - not syncing: LBUG [216597.870992] CPU: 11 PID: 29045 Comm: lfsck Tainted: G           OE  ------------   3.10.0-693.21.1.el7.20180508.x86_64.lustre2105 #1 [216597.870994] Call Trace: [216597.870999]  [&lt;ffffffff8168f4b8&gt;] dump_stack+0x19/0x1b [216597.871003]  [&lt;ffffffff81689aa2&gt;] panic+0xe8/0x21f [216597.871008]  [&lt;ffffffffa086d8cb&gt;] lbug_with_loc+0x9b/0xa0 [libcfs] [216597.871017]  [&lt;ffffffffa10e06fd&gt;] lfsck_layout_slave_prep+0x51d/0x590 [lfsck] [216597.871025]  [&lt;ffffffffa10ba624&gt;] lfsck_master_engine+0x184/0x1360 [lfsck] [216597.871032]  [&lt;ffffffffa10ba4a0&gt;] ? lfsck_master_oit_engine+0x1190/0x1190 [lfsck] [216597.871034]  [&lt;ffffffff810b1131&gt;] kthread+0xd1/0xe0 [216597.871036]  [&lt;ffffffff810b1060&gt;] ? insert_kthread_work+0x40/0x40 [216597.871038]  [&lt;ffffffff816a14f7&gt;] ret_from_fork+0x77/0xb0 [216597.871039]  [&lt;ffffffff810b1060&gt;] ? insert_kthread_work+0x40/0x40 [216597.871698] LustreError: 29049:0:(lfsck_layout.c:5144:lfsck_layout_slave_prep()) ASSERTION( !llsd-&gt;llsd_rbtree_valid ) failed: [216597.871700] LustreError: 29049:0:(lfsck_layout.c:5144:lfsck_layout_slave_prep()) LBUG [216597.871701] Pid: 29049, comm: lfsck 3.10.0-693.21.1.el7.20180508.x86_64.lustre2105 #1 SMP Mon Aug 27 23:04:41 UTC 2018 [216597.871702] Call Trace: [216597.871710]  [&lt;ffffffff8103a1f2&gt;] save_stack_trace_tsk+0x22/0x40 [216597.871716]  [&lt;ffffffffa086d7cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [216597.871721]  [&lt;ffffffffa086d87c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [216597.871733]  [&lt;ffffffffa10e06fd&gt;] lfsck_layout_slave_prep+0x51d/0x590 [lfsck] [216597.871741]  [&lt;ffffffffa10ba624&gt;] lfsck_master_engine+0x184/0x1360 [lfsck] [216597.871743]  [&lt;ffffffff810b1131&gt;] kthread+0xd1/0xe0 [216597.871745]  [&lt;ffffffff816a14f7&gt;] ret_from_fork+0x77/0xb0 </pre> </div></div> <p> </p> <p>I hope this helps, and I have already uploaded the complete stack trace to FTP site.</p> <p>Thank You,</p> <p>                  Manish </p> <p> </p> <p> </p> <p>Hi Lai,</p> <p> </p> <p>Just wanted to follow up, any updates on this issues, as we will be getting the patch from other ticket <a href="https://jira.whamcloud.com/browse/LU-11584" title="kernel BUG at ldiskfs.h:1907!" class="issue-link" data-issue-key="LU-11584"><del>LU-11584</del></a> for lfsck and if this issues can be addressed in that patch then it would be nice to avoid one more outage because of this bug.</p> <p>Thank You,</p> <p>                  Manish</p> <p>I'm still reviewing code to understand this, and it doesn't look to be the same issue of <a href="https://jira.whamcloud.com/browse/LU-11584" title="kernel BUG at ldiskfs.h:1907!" class="issue-link" data-issue-key="LU-11584"><del>LU-11584</del></a>.</p> <p>Manish, do you remember how you started 'lfsck' on servers? Did you only run 'lfsck' on all OSS? but not MDS? And what's the exact command?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lctl lfsc_start -o -r </pre> </div></div> <p>This likely related to <a href="https://jira.whamcloud.com/browse/LU-11625" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-11625</a></p> <p>Mahmoud, does your system have multiple MDTs? Did you run 'lctl lfsck_start -o -r' on MDS either?</p> <p>We just hit this one on 2.10.8. <br/> I was running `lctl lfsck_start -M MDT-0000 -o -r` on MDS0</p> <p>We have 2 MDS - 2 MDT and we are running 1 MDT on each MDS. </p> Development Rank 1|i005gn: Rank (Obsolete) 9223372036854775807 Severity