https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11082 Lustre <p>Hi,</p> <p>our main filesystem has hung approximately 10 times since April.</p> <p>the symptoms are stuck thread messages on the MDS (it has happened on both our MDS's), and then soon afterwards clients can't stat any files in the filesystem.</p> <p>we have 4 filesystems that share the same MDS hardware and only the large DNE filesystem with 3 MDTs has hung, so we think this is probably a DNE issue.</p> <p>there are no logs that indicate this is a MDS hardware problem. despite that we've swapped out a lot of hardware and tested the MDS components as well as we can and it hasn't helped at all. 'zpool iostat 2' shows some tiny amounts of i/o are still going to the MDTs when the filesystems are hung, so eg. hardware raid isn't just silently stopping with no logs.</p> <p>MDT's are zmirror on top of hardware raid1 (so 4-way replicated).</p> <p>to un-hang the filesystem, we failover or reboot the MDS. every time we can't umount one of the 3 MDTs in the /dagg(fred) filesystem, and we have to forcibly power off the MDS (which led to LU10877 etc.). recovery these days goes ok, and then the filesystem works again. it seems random which of the three MDTs hangs on umount. we've lfsck'd the namespace about 5 times.</p> <p>here are first stack traces from each of the 2 separate hangs today, which are pretty typical. usually it's a mdt_rdpg* thread that hangs and those stack traces all look very similar, but occasionally it's a mdt* thread and there's some variations on a theme in those.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>/var/log/messages:Jun 11 11:06:59 warble2 kernel: LNet: Service thread pid 116276 was inactive for 200.63s. The thread might be hung, or it might only be slow and will res ume later. Dumping the stack trace for debugging purposes: /var/log/messages:Jun 11 11:06:59 warble2 kernel: Pid: 116276, comm: mdt_rdpg00_024 /var/log/messages:Jun 11 11:06:59 warble2 kernel: #012Call Trace: /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffff8109eb8b&gt;] ? recalc_sigpending+0x1b/0x50 /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffff816b40e9&gt;] schedule+0x29/0x70 /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffffc0e16f10&gt;] top_trans_wait_result+0xa6/0x155 [ptlrpc] /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffff810c7c70&gt;] ? default_wake_function+0x0/0x20 /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffffc0df882b&gt;] top_trans_stop+0x42b/0x930 [ptlrpc] /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffffc164f5e9&gt;] lod_trans_stop+0x259/0x340 [lod] /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffffc0df8fae&gt;] ? top_trans_start+0x27e/0x940 [ptlrpc] /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffffc16ed1ea&gt;] mdd_trans_stop+0x2a/0x46 [mdd] /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffffc16e2bcb&gt;] mdd_attr_set+0x5eb/0xce0 [mdd] /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffffc0d81d47&gt;] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffffc15ce186&gt;] mdt_mfd_close+0x1a6/0x610 [mdt] /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffffc15d3901&gt;] mdt_close_internal+0x121/0x220 [mdt] /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffffc15d3c20&gt;] mdt_close+0x220/0x780 [mdt] /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffffc0de52ba&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffffc0d8de2b&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffffc0d8a458&gt;] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffff810c7c82&gt;] ? default_wake_function+0x12/0x20 /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffff810bdc4b&gt;] ? __wake_up_common+0x5b/0x90 /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffffc0d91572&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffffc0d90ae0&gt;] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffff810b4031&gt;] kthread+0xd1/0xe0 /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffff810b3f60&gt;] ? kthread+0x0/0xe0 /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffff816c055d&gt;] ret_from_fork+0x5d/0xb0 /var/log/messages:Jun 11 11:06:59 warble2 kernel: [&lt;ffffffff810b3f60&gt;] ? kthread+0x0/0xe0 /var/log/messages:Jun 11 11:06:59 warble2 kernel: /var/log/messages:Jun 11 11:06:59 warble2 kernel: LustreError: dumping log to /tmp/lustre-log.1528679219.116276 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jun 11 14:17:28 warble2 kernel: LNet: Service thread pid 22072 was inactive for 200.35s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jun 11 14:17:28 warble2 kernel: Pid: 22072, comm: mdt00_001 Jun 11 14:17:28 warble2 kernel: #012Call Trace: Jun 11 14:17:28 warble2 kernel: [&lt;ffffffff8109eb8b&gt;] ? recalc_sigpending+0x1b/0x50 Jun 11 14:17:28 warble2 kernel: [&lt;ffffffff816b40e9&gt;] schedule+0x29/0x70 Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc0dd3f10&gt;] top_trans_wait_result+0xa6/0x155 [ptlrpc] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffff810c7c70&gt;] ? default_wake_function+0x0/0x20 Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc0db582b&gt;] top_trans_stop+0x42b/0x930 [ptlrpc] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc10f35e9&gt;] lod_trans_stop+0x259/0x340 [lod] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc0db5fae&gt;] ? top_trans_start+0x27e/0x940 [ptlrpc] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc11911ea&gt;] mdd_trans_stop+0x2a/0x46 [mdd] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc117eb31&gt;] mdd_rename+0x4d1/0x14a0 [mdd] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc103b07a&gt;] mdt_reint_rename_internal.isra.36+0x166a/0x20c0 [mdt] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc103d3fb&gt;] mdt_reint_rename_or_migrate.isra.39+0x19b/0x860 [mdt] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc0d0eff0&gt;] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc0d09440&gt;] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc103daf3&gt;] mdt_reint_rename+0x13/0x20 [mdt] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc1041b33&gt;] mdt_reint_rec+0x83/0x210 [mdt] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc102337b&gt;] mdt_reint_internal+0x5fb/0x9c0 [mdt] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc102ef07&gt;] mdt_reint+0x67/0x140 [mdt] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc0da22ba&gt;] tgt_request_handle+0x92a/0x1370 [ptlrpc] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc0d4ae2b&gt;] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc0d47458&gt;] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffff810c7c82&gt;] ? default_wake_function+0x12/0x20 Jun 11 14:17:28 warble2 kernel: [&lt;ffffffff810bdc4b&gt;] ? __wake_up_common+0x5b/0x90 Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc0d4e572&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffffc0d4dae0&gt;] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Jun 11 14:17:28 warble2 kernel: [&lt;ffffffff810b4031&gt;] kthread+0xd1/0xe0 Jun 11 14:17:28 warble2 kernel: [&lt;ffffffff810b3f60&gt;] ? kthread+0x0/0xe0 Jun 11 14:17:28 warble2 kernel: [&lt;ffffffff816c055d&gt;] ret_from_fork+0x5d/0xb0 Jun 11 14:17:28 warble2 kernel: [&lt;ffffffff810b3f60&gt;] ? kthread+0x0/0xe0 Jun 11 14:17:28 warble2 kernel: Jun 11 14:17:28 warble2 kernel: LustreError: dumping log to /tmp/lustre-log.1528690648.22072 </pre> </div></div> <p>I've attach logs of all the stack traces</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>zgrep warble /var/log/messages-20180[3456]* /var/log/messages | egrep -v 'crmd|pengine' | grep -i 'stack trace' -B20 -A40 &gt; ~/lustre-bugs/warble-stack-traces.txt </pre> </div></div> <p>clients at the moment are a mix of centos7.4 and 2.10.3/2.10.4, and centos7.5 2.10.4.<br/> servers are centos7.4 and 2.10.4 with zfs 0.7.9 and boot with nopti.</p> <p>we have inherited directory striping on the main filesystem, so each new dir is on a random one of the three MDTs.</p> <p>we have no idea what i/o pattern triggers the lockups, sorry.</p> <p>cheers,<br/> robin</p> x86_64 centos7.4 OPA ZFS 0.7.9 lustre 2.10.4 LU-11082

stuck threads on MDS

Bug Minor Resolved Fixed Lai Siyao SC Admin Mon, 11 Jun 2018 16:02:36 +0000 Thu, 16 Aug 2018 18:55:12 +0000 Thu, 16 Aug 2018 18:55:11 +0000 Lustre 2.10.3 Lustre 2.10.4 0 6 <p>Lai</p> <p>Can you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>I think rename is the culprit, because I've seen such deadlock in other tests. The root cause is that 'rename' needs to lock both source and target parents, and there are two cases it should lock them in reverse order to avoid deadlock:<br/> 1. source parent is a child of target parent.<br/> 2. both source and target parents are stripes of a striped directory, and the stripe index of source parent is after that of target parent.</p> <p>Now 'rename' only checks the first situation, I've cooked a fix under ticket <a href="https://jira.whamcloud.com/browse/LU-4684" title="DNE3: allow migrating DNE striped directory" class="issue-link" data-issue-key="LU-4684"><del>LU-4684</del></a>: <a href="https://review.whamcloud.com/#/c/32701/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/32701/</a>. But it's not fully tested and reviewed yet, you can wait for it to be ready.</p> <p>that sounds plausible for sure.</p> <p>this is an important issue to us. the cluster is broken when the main filesystem hangs.<br/> can we raise the priority of this somehow?</p> <p>cheers,<br/> robin</p> <p>Robin</p> <p>Don't worry we are treating this as a priority. However, experience has shown that any changes to the DNE code need to be very thoroughly tested and this includes multiple days of stress testing. We will definitely advise as soon as we are confident that the fix does not have any undesired side-effects. </p> <p>Peter</p> <p>coolio. please let us know if you'd like us to try something out. I don't mind if things crash, as long as it's contributing to progress.</p> <p>cheers,<br/> robin</p> <p>Hiya,</p> <p>5 more reboots of the MDS's today.</p> <p>this time they didn't stop until a script that was doing find and then chgrp/chmod on a client was killed.</p> <p>this process does no mv or rename so doesn't seem to fit in with your current theory of why the MDS threads hang.</p> <p>I'll attach the trace from the first hang on the MDS.</p> <p>cheers,<br/> robin</p> <p>Hiya,</p> <p>now that I think about it, I'm pretty sure the find/chgrp/chmod script was also running in April when we first hit MDT hangs (and then <a href="https://jira.whamcloud.com/browse/LU-10887" title="2 MDTs stuck in WAITING" class="issue-link" data-issue-key="LU-10887"><del>LU-10887</del></a> as fallout), so it might be a reproducer for this. presumably users might also have been doing similar things to cause the in-between hangs.</p> <p>this script is on 1 client and runs ~5 of the below at once, each on separate directories</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>( find $p -mount ! -gid $gid -ls -exec chgrp -h $g {} \; -type d -exec chmod g+s {} \; ; find $p -mount -type d ! -perm -g=s -exec chmod g+s {} \; -ls ) </pre> </div></div> <p>it forces file and dir group ownership to match the project directory name, and sets g+s on dirs. ie. a poor mans project/directory quotas. it's a pretty common thing for sites to want to do.</p> <p>the script itself is a bit more complicated than the above, with some VFS stuff to try to limit inode lists getting too long on the sweeping client and slowing down the kernel, but the above line is the essence of it. let me know if you want the full thing.</p> <p>cheers,<br/> robin</p> <p>Hmm, it looks like there are more deadlocks than rename, I'm looking into them. Thanks!</p> <p>I just uploaded three patches, I'd suggest you apply them all and then test again:<br/> <a href="https://review.whamcloud.com/#/c/32589/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/32589/</a><br/> <a href="https://review.whamcloud.com/#/c/32738/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/32738/</a><br/> <a href="https://review.whamcloud.com/#/c/32701/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/32701/</a></p> <p>Hi,</p> <p>any chance you can make a version of <a href="https://review.whamcloud.com/#/c/32589/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/32589/</a> for 2.10?<br/> it doesn't apply at the moment.</p> <p>cheers,<br/> robin</p> <p>Hi,</p> <p>how's this coming along?<br/> there are 1000's of lines of code different between b2_10 and master across these 4 files, so I'm not confident about backporting this myself to try it out.</p> <p>cheers,<br/> robin</p> <p>The backport for 2.10 of <a href="https://review.whamcloud.com/#/c/32589/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/32589/</a> is on <a href="https://review.whamcloud.com/32853" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32853</a>.</p> <p>thanks. we have that on out MDS's now.</p> <p>cheers,<br/> robin</p> <p>Robin</p> <p>How long will this patch need to be running before you would conclude that it has improved the situation?</p> <p>Peter</p> <p>Hi Peter,</p> <p>no issues so far.</p> <p>however I haven't yet run the chmod/chgrp sweep script which was maybe a trigger for this issue.<br/> partly because (TBH) I've been enjoying the stability, but also because I haven't had any free time.<br/> I'll run it this week and report back.</p> <p>cheers,<br/> robin</p> <p>Hi,</p> <p>after 2 days of running, the find/chgrp/chmod sweep through all files almost completed, but not quite.</p> <p>once again the MDTs hung, but it looked different to the previous hangs. this time all 3 MDTs wouldn't umount, and previously it's just been (a random) one of them. resolution was the same though - had to power cycle the MDS.</p> <p>I'll attach all warble related syslog (warble1 is where all 3 MDTs are mounted at the moment).<br/> I'll attach the first lustre-log.1533268164.11346.gz.</p> <p>stack traces were not printed to anywhere because</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Aug 3 13:49:24 warble1 kernel: LNet: 17382:0:(linux-debug.c:185:libcfs_call_trace()) can't show stack: kernel doesn't export show_task </pre> </div></div> <p>is this a new change?</p> <p>we're running 3.10.0-862.9.1.el7.x86_64 centos7.5 everywhere now. server lustre is 2.10.4 plus -&gt;</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> lu10683-lu11093-checksum-overquota-gerrit32788-1fb85e7e.patch lu10988-lfsck2-gerrit32522-21d33c11.patch lu11074-mdc-xattr-gerrit32739-dea1cde9.patch lu11107-xattr-gerrit32753-c96a8f08.patch lu11111-lfsck-gerrit32796-693fe452.patch lu11082-lu11103-stuckMdtThreads-gerrit32853-3dc08caa.diff </pre> </div></div> <p>because the symptoms are different, if I had to guess I'd say that the patch here has helped a lot, and perhaps fixed the issue, and maybe we're running into something new now, but without stacktraces I really have no idea.</p> <p>cheers,<br/> robin</p> <p>Is the stack trace issue <a href="https://jira.whamcloud.com/browse/LU-11062" title="Backtrace stack printing is broken in RHEL 7.5" class="issue-link" data-issue-key="LU-11062"><del>LU-11062</del></a> perhaps?</p> <p>Hi Peter,</p> <p>yes, that looks like it. thanks.<br/> I've included the b2_10 version of the stacktrace patch into our MDS builds, and kicked off a re-run of the chmod/chgrp script.</p> <p>cheers,<br/> robin</p> <p>the chgrp/chmod script finished successfully. fixed up about 10M files and a lot of dirs over 2.5 days. no crash on the MDS.</p> <p>as the hung threads in the previous crash looked different to those at the start of this ticket, and this most recent sweep finished without issue, I would err on the side of saying this issue has been fixed.</p> <p>thanks!</p> <p>cheers,<br/> robin</p> <p>Good news. So let's close out this ticket then</p> Related LU-11103 LU-11104 LU-11102 Development Rank 1|hzzy8f: Rank (Obsolete) 9223372036854775807 Severity