https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11062 Lustre <p>It looks like struct stacktrace_ops is no longer there in rhel7.5 so our detection of if dump_stack would work is no longer working.</p> <p>We get this now:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[29185.979935] LNet: Service thread pid 9373 was inactive for 40.04s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [29185.981691] LNet: 31237:0:(linux-debug.c:185:libcfs_call_trace()) can't show stack: kernel doesn't export show_task [29185.982826] LustreError: dumping log to /tmp/lustre-log.1527605717.9373 </pre> </div></div> <p>Given that struct stacktrace_ops are still there in mainline kernels, this seems something specific to rhel7.5 and we need to find another way of detecting this I guess.</p> LU-11062

Backtrace stack printing is broken in RHEL 7.5

Bug Major Resolved Fixed Yang Sheng Oleg Drokin Tue, 29 May 2018 16:56:21 +0000 Thu, 13 Sep 2018 12:40:50 +0000 Mon, 13 Aug 2018 13:58:08 +0000 Lustre 2.12.0 Lustre 2.10.5 0 10 <p>I may be misreading it, but it looks to me like this only affects cases where one thread (e.g. watchdog) is attempting to dump the stack of another thread, and stacktrace_ops/print_trace_ops are used.</p> <p>In the case where a thread is dumping its own thread, dump_stack() is called, and still works, I believe.</p> <p>Yang Sheng</p> <p>Could you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>Whomever is working on this, please also update the error message to print <tt>kernel doesn't export dump_trace()</tt> since <tt>show_stack()</tt> hasn't existed for a long time.</p> <p>It looks like the relevant kernel commit is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit e18bcccd1a4ecb41e99678e002ef833586185bf1 Author: Josh Poimboeuf &lt;jpoimboe@redhat.com&gt; AuthorDate: Fri Sep 16 14:18:16 2016 -0500 Commit: Ingo Molnar &lt;mingo@kernel.org&gt; CommitDate: Tue Sep 20 08:29:34 2016 +0200 x86/dumpstack: Convert show_trace_log_lvl() to use the new unwinder Convert show_trace_log_lvl() to use the new unwinder. dump_trace() has been deprecated. </pre> </div></div> <p>This was disabled in Lustre via:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 70d70c4f541c84adc88c372d670cd3a7fa8bc91f Author: Dmitry Eremin &lt;dmitry.eremin@intel.com&gt; LU-9183 libcfs: handle dump_trace() and related callbacks removal In kernel version 4.8 commit c8fe4609827aedc9c4b45de80e7cdc8ccfa8541b all previous users of dump_trace() have been converted to use the new unwind interfaces, so the dump_trace() and the related print_context_stack() and print_context_stack_bp() callback functions were removed. Change-Id: Ifa7a112d622b23f733d6daab05f9838afdf31a86 Reviewed-on: https://review.whamcloud.com/25816 </pre> </div></div> <p>It seems we need to update to the new kernel API, since this was backported to the RHEL7.5 kernel.</p> <p>Maybe its time to look at using the hang checker available in the kernel?</p> <p>James, do you have any pointers to the kernel mechanism? We already get NMI watchdogs from the kernel, but those are when the thread doesn't schedule for a long time. Our current code can dump the stack on another service thread that is not making progress, even though it isn't totally dead (e.g. in a sleep/check loop). The thread can "ping" the watchdog to tell it is is alive, and lack of pings == lack of progress. If the kernel can do something similar, especially if it's been around a while, then I'd be happy to see it.</p> <p>hangcheck timer reports on idle threads that did not schedule in a while or some such (it needs to be compiled in and enabled), but it's not quite what we need since our timeouts are different and we have a bit different model of when to start the countdown and when to stop it.</p> <p>I was talking to Neil and he recommend we use save_stack_trace_tsk() since it been around since 2.6.29.</p> <p>Also of note Shadow pointed out that the kernel does have a soft lockup thread (CONFIG_LOCKUP_DETECTOR). It appears enabled for RHEL kernels. Their is also a detect hung_task functionality.</p> <p>FWIW we got hung tasks in <a href="https://jira.whamcloud.com/browse/LU-11082" title="stuck threads on MDS" class="issue-link" data-issue-key="LU-11082"><del>LU-11082</del></a> and got this message and no stack traces. in retrospect I guess I should have tried sysrq to get stack traces before rebooting. I've got used to Lustre doing it for me.</p> <p>cheers,<br/> robin</p> <p>Yang Sheng (ys@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/32952" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32952</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11062" title="Backtrace stack printing is broken in RHEL 7.5" class="issue-link" data-issue-key="LU-11062"><del>LU-11062</del></a> libcfs: use save_stack_trace for stack dump<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bc344d10030aa60385f0577a855993733b38c916</p> <p>I have submitted a patch for this ticket. Just use save_stack_trace_tsk for backtrace dump. The obvious problem is that unable to judge a address whether reliable or not. This will make things confused in some case.</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/32972" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32972</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11062" title="Backtrace stack printing is broken in RHEL 7.5" class="issue-link" data-issue-key="LU-11062"><del>LU-11062</del></a> libcfs: use save_stack_trace for stack dump<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 7232c445fe30f6500f6f731ef8ffad617490eb68</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32952/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32952/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11062" title="Backtrace stack printing is broken in RHEL 7.5" class="issue-link" data-issue-key="LU-11062"><del>LU-11062</del></a> libcfs: use save_stack_trace for stack dump<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: afedf9343686504c89f2e28cf6133540166f2347</p> <p>Landed for 2.12</p> <p>John L. Hammond (jhammond@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32972/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32972/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11062" title="Backtrace stack printing is broken in RHEL 7.5" class="issue-link" data-issue-key="LU-11062"><del>LU-11062</del></a> libcfs: use save_stack_trace for stack dump<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: a2af371bd8a79e293a9ba95b8016de92040101a6</p> Related LU-9183 LU-11241 Development Rank 1|hzzxxj: Rank (Obsolete) 9223372036854775807 Severity