https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6173 Lustre <p>Yesterday experienced a network problem. Consequently, we had a number of clients stalled. At least four were hanged in this situation. We captured a vmcore on one of the systems.</p> <p>Console logs showed one of the CPUs was detected to stall:<br/> "INFO: rcu_sched_state detected stall on CPU 9."</p> <p>All CPU's at r305i7n2 except CPU 9 were running migration process and<br/> the rcu_sched_state detected CPU was running obd_zombid.<br/> The console logs of other three systems confirmed the stalled CPU were<br/> running obd_zombid also, but without vmcore I can not say for sure that<br/> other CPU's were running 'migration' as r305i7n2 did.</p> <p>The stack trace is:</p> <p>PID: 5070 TASK: ffff88046f086300 CPU: 9 COMMAND: "obd_zombid"<br/> #0 <span class="error">&#91;ffff88087fc27e40&#93;</span> crash_nmi_callback at ffffffff810245af<br/> #1 <span class="error">&#91;ffff88087fc27e50&#93;</span> notifier_call_chain at ffffffff81475847<br/> #2 <span class="error">&#91;ffff88087fc27e80&#93;</span> __atomic_notifier_call_chain at ffffffff8147588d<br/> #3 <span class="error">&#91;ffff88087fc27e90&#93;</span> notify_die at ffffffff814758dd<br/> #4 <span class="error">&#91;ffff88087fc27ec0&#93;</span> default_do_nmi at ffffffff81472d37<br/> #5 <span class="error">&#91;ffff88087fc27ee0&#93;</span> do_nmi at ffffffff81472f68<br/> #6 <span class="error">&#91;ffff88087fc27ef0&#93;</span> restart_nmi at ffffffff814724b1<br/> <span class="error">&#91;exception RIP: native_halt+1&#93;</span><br/> RIP: ffffffff810300b1 RSP: ffff88087fc23de0 RFLAGS: 00000082<br/> RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000080f<br/> RDX: 0000000000000000 RSI: 00000000000000ff RDI: 000000000000080f<br/> RBP: ffff88046d96fd78 R8: 0000000000000150 R9: ffffe8ffffc20738<br/> R10: 0000000000000006 R11: ffffffff8102b430 R12: 0000000000000000<br/> R13: 0000000000000006 R14: 0000000000000006 R15: 00000000fffffffb<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> &#8212; &lt;NMI exception stack&gt; &#8212;<br/> #7 <span class="error">&#91;ffff88087fc23de0&#93;</span> native_halt at ffffffff810300b1<br/> #8 <span class="error">&#91;ffff88087fc23de0&#93;</span> halt_current_cpu at ffffffff81024959<br/> #9 <span class="error">&#91;ffff88087fc23df0&#93;</span> lkdb_main_loop at ffffffff812548ec<br/> #10 <span class="error">&#91;ffff88087fc23ef0&#93;</span> kdba_main_loop at ffffffff8139bef2<br/> #11 <span class="error">&#91;ffff88087fc23f20&#93;</span> kdb at ffffffff8125199f<br/> #12 <span class="error">&#91;ffff88087fc23f80&#93;</span> kdb_ipi at ffffffff8124ea07<br/> #13 <span class="error">&#91;ffff88087fc23f90&#93;</span> smp_kdb_interrupt at ffffffff8139b656<br/> #14 <span class="error">&#91;ffff88087fc23fb0&#93;</span> kdb_interrupt at ffffffff8147aca3<br/> &#8212; &lt;IRQ stack&gt; &#8212;<br/> #15 <span class="error">&#91;ffff88046d96fd78&#93;</span> kdb_interrupt at ffffffff8147aca3<br/> <span class="error">&#91;exception RIP: _raw_spin_lock+24&#93;</span><br/> RIP: ffffffff81471a88 RSP: ffff88046d96fe28 RFLAGS: 00000206<br/> RAX: 0000000000001700 RBX: ffff880867d28810 RCX: ffff880856c3be00<br/> RDX: 0000000000008000 RSI: ffff880856c3be00 RDI: ffff880430b100f8<br/> RBP: ffff880864634078 R8: 0000000000000002 R9: 0000000000000000<br/> R10: 0000000010000008 R11: 0000000000000000 R12: ffffffff8147ac9e<br/> R13: ffffffff811458be R14: ffff880867d28810 R15: 0000000000000206<br/> ORIG_RAX: ffffffffffffff01 CS: 0010 SS: 0018<br/> #16 <span class="error">&#91;ffff88046d96fe28&#93;</span> osc_cleanup at ffffffffa0a48829 <span class="error">&#91;osc&#93;</span><br/> #17 <span class="error">&#91;ffff88046d96fe38&#93;</span> class_decref at ffffffffa076eed4 <span class="error">&#91;obdclass&#93;</span><br/> #18 <span class="error">&#91;ffff88046d96fea8&#93;</span> class_export_destroy at ffffffffa074c1de <span class="error">&#91;obdclass&#93;</span><br/> #19 <span class="error">&#91;ffff88046d96fec8&#93;</span> obd_zombie_impexp_cull at ffffffffa074c61d <span class="error">&#91;obdclass&#93;</span><br/> #20 <span class="error">&#91;ffff88046d96fee8&#93;</span> obd_zombie_impexp_thread at ffffffffa074c7bd <span class="error">&#91;obdclass&#93;</span><br/> #21 <span class="error">&#91;ffff88046d96ff48&#93;</span> kernel_thread_helper at ffffffff8147aae4</p> Git repo can be found at <a href="https://github.com/jlan/lustre-nas">https://github.com/jlan/lustre-nas</a><br/> Server: centos 6.4 2.6.32-358.23.2.el6, lustre 2.4.3-12nasS<br/> Client: sles11sp3 3.0.101-0.31.1, lustre 2.4.3-11nasC LU-6173

CPU stalled with obd_zombid running

Bug Major Resolved Fixed Emoly Liu Jay Lan Wed, 28 Jan 2015 23:52:06 +0000 Thu, 14 Jun 2018 21:41:37 +0000 Mon, 25 May 2015 22:41:49 +0000 Lustre 2.7.0 Lustre 2.4.3 Lustre 2.5.3 Lustre 2.8.0 0 7 <p>Emoly</p> <p>Could you please advise?</p> <p>Peter</p> <p>Jay, could you please upload the full vmcore file for further investigation? Thanks.</p> <p>BTW, do you have the dmesg log? I want to know what happened during the network problem.</p> <p>File r305i7n2-20150128.bz2 contains the console log of the system when the system getting into trouble. Since stack traces of all CPU's were printed every 10 minutes, the network errors messages were flushed out from the demsg buffer, so I attached the console log here instead.</p> <p>User ID in the log has been replaced with 'xxx'.</p> <p>Vmcore can only be seen by US citizen. I can send crash analysis information to you if that is OK. Otherwise, I will consult with other guys on how to send you encrypted vmcore. Please advise.</p> <p>Jay, I am not a US citizen. You can send the crash analysis information to me first. If necessary, I will ask for my other US citizen colleague to help.</p> <p>This tarball contains output of 'bt -a', 'ps -a', 'kmem -i' and 'kmem -s'.</p> <p>Let me know if you want me to provide you output of other crash commands.</p> <p>Hm, this really reminds me of <a href="https://jira.whamcloud.com/browse/LU-2543" title="obd_zombid oops" class="issue-link" data-issue-key="LU-2543"><del>LU-2543</del></a> that I was hitting in pre 2.4 times, but we had a pretty effective patch for that that you do have in your tree.<br/> I wonder if there was another corner case left that was closed later.</p> <p>Slightly different. <a href="https://jira.whamcloud.com/browse/LU-2543" title="obd_zombid oops" class="issue-link" data-issue-key="LU-2543"><del>LU-2543</del></a> resulted in a kernel panic while we experienced system hang in this case.</p> <p>Well, you see - I hit kernel panic in <a href="https://jira.whamcloud.com/browse/LU-2543" title="obd_zombid oops" class="issue-link" data-issue-key="LU-2543"><del>LU-2543</del></a> because I was running with a special kernel with a lot of extra debug turned on. So every time there's access to freed memory - it'l crash, or if there's invalid spinlock magic - It'll crash and so on.<br/> Those options are very heavy weight, so nobody runs with them in production.<br/> So in your kernel if you encounter such a spinlock with random garbage in memory (there is not even a magic in the non-debug code so that it could be faster) - it'll just take this garbage at face value and will wait until the other holed (imagined) will go away and since there's nobody actually there holding tha tspinlock - the code will wait there forever - this is what we see in your backtrace.</p> <p>And that's why I think it's something very similar.</p> <p>I ever suspected <a href="https://jira.whamcloud.com/browse/LU-2543" title="obd_zombid oops" class="issue-link" data-issue-key="LU-2543"><del>LU-2543</del></a> too, but its patch has been included. This issue needs more investigation.</p> <p>Yes, it does need more investigation, no question about that.<br/> It does look like this is another case of use after free to me.<br/> I need lustre kernel modules with debug info from your build to get some more info, please.</p> <p>Hi Oleg,</p> <p>The lustre client debuginfo rpm has been uploaded to ftp.whamcloud.com. I attached ".<a href="https://jira.whamcloud.com/browse/LU-6173" title="CPU stalled with obd_zombid running" class="issue-link" data-issue-key="LU-6173"><del>LU-6173</del></a>" to the end of the rpm file.</p> <p>So, poking around in the crashdump, it looks like it is indeed something very similar to <a href="https://jira.whamcloud.com/browse/LU-2543" title="obd_zombid oops" class="issue-link" data-issue-key="LU-2543"><del>LU-2543</del></a>.<br/> What we are seeing in the log is that the filesystem with name nbp6 is being unmounted while there are some communication problems to its OSTs (probably network hiccup mentioned).<br/> By the time the crash happened the umount of nbp6 was already unmounted and the sbi structure freed, but OSC cleanups are still ongoing and those do access content of the sbi struct (ll_cache member of it). Since it contains garbage, attempt to geta spinlock fails.<br/> This is evident since the only two left lustre filesystems mounted are nbp5 and nbp9.</p> <p>So, examining the disconnect code, it looks like client_common_put_super assumes the mere call to obd_disconnect(sbi-&gt;ll_dt_exp); just marks the import disconnected, but if there are any requests in flight (highly likely if you have a broken connection and requests take seconds to timeout), then the actual final import put would not happen until this last request is finished (every request holds an import reference), and only then the final class_import_put() would happen that would call obd_zombie_import_add() increasing the zombie task list count and would stall obd_zombie_barrier().</p> <p>So the "fix" for <a href="https://jira.whamcloud.com/browse/LU-2543" title="obd_zombid oops" class="issue-link" data-issue-key="LU-2543"><del>LU-2543</del></a> really failed to consider this scenario of inflight requests for all imports.<br/> I see that ll_cache itself has a refcounter inside it, and perhaps that might be a much better proxy to determine when we are fine freeing the sbi struct. Niu?</p> <p>Actually I guess that would lead to unmount hanging until all requests finish processing which might not be ideal either in the face of broken connection, so potentially sbi freeing could be made asynchronous too.<br/> This bug exists in master too, btw.</p> <blockquote> <p>So, examining the disconnect code, it looks like client_common_put_super assumes the mere call to obd_disconnect(sbi-&gt;ll_dt_exp); just marks the import disconnected, but if there are any requests in flight (highly likely if you have a broken connection and requests take seconds to timeout), then the actual final import put would not happen until this last request is finished (every request holds an import reference), and only then the final class_import_put() would happen that would call obd_zombie_import_add() increasing the zombie task list count and would stall obd_zombie_barrier().<br/> So the "fix" for <a href="https://jira.whamcloud.com/browse/LU-2543" title="obd_zombid oops" class="issue-link" data-issue-key="LU-2543"><del>LU-2543</del></a> really failed to consider this scenario of inflight requests for all imports.</p></blockquote> <p>Will the inflight RPC hold the OSC export refcount as well? I was thinking that obd_disconnect() in client_common_put_super() shall put the last refcount of OSC export and make the umount wait in obd_zombie_barrier().</p> <p>Niu: It's right in the __ptlrpc_request_alloc():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> request-&gt;rq_import = class_import_get(imp); </pre> </div></div> <p>and the import stays put until all requests are drained, which might take awhile if the requests are stuck on the network.</p> <p>Emoly Liu (emoly.liu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13727" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13727</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6173" title="CPU stalled with obd_zombid running" class="issue-link" data-issue-key="LU-6173"><del>LU-6173</del></a> llite: allocate and free client cache asynchronously<br/> Project: fs/lustre-release<br/> Branch: b2_4<br/> Current Patch Set: 1<br/> Commit: ae23e1e99d072c3865ca2da538705eb61fc6c7c2</p> <p>Thanks for Niu&amp;Oleg's help! I pushed a patch for b2_4 for review.</p> <p>Emoly</p> <p>Is this patch also required for master/b2_5?</p> <p>Peter</p> <p>Peter, yes both master and b2_5 need the patch. I will create one for master later.</p> <p>Emoly Liu (emoly.liu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13746" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13746</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6173" title="CPU stalled with obd_zombid running" class="issue-link" data-issue-key="LU-6173"><del>LU-6173</del></a> llite: allocate and free client cache asynchronously<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 570a48915a6935b8d180dafded4befaa2447b585</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13746/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13746/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6173" title="CPU stalled with obd_zombid running" class="issue-link" data-issue-key="LU-6173"><del>LU-6173</del></a> llite: allocate and free client cache asynchronously<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 302c5bfebe61e988dbd27063becc4ef90befc6df</p> <p>Landed for 2.8</p> <p>Could you provide a 2.5 back port? Thanks!</p> <p>Yes this is being worked on</p> Related Development Rank 1|hzx55b: Rank (Obsolete) 17274 Severity