[LU-11062] Backtrace stack printing is broken in RHEL 7.5

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.12.0, Lustre 2.10.5
• Type: Bug
• Priority: Major
• Reporter: Oleg Drokin
• Assignee: Yang Sheng
• Resolution: Fixed
• Severity: 3

Description

It looks like struct stacktrace_ops is no longer there in rhel7.5 so our detection of if dump_stack would work is no longer working.

We get this now:

[29185.979935] LNet: Service thread pid 9373 was inactive for 40.04s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[29185.981691] LNet: 31237:0:(linux-debug.c:185:libcfs_call_trace()) can't show stack: kernel doesn't export show_task
[29185.982826] LustreError: dumping log to /tmp/lustre-log.1527605717.9373

Given that struct stacktrace_ops are still there in mainline kernels, this seems something specific to rhel7.5 and we need to find another way of detecting this I guess.

Issue Links

• is related to: LU-9183 Support for linux 4.9 kernels
• is related to: LU-11241 Unable to umount during MDT fail back

Comments

[Olaf Faaland]

I may be misreading it, but it looks to me like this only affects cases where one thread (e.g. watchdog) is attempting to dump the stack of another thread, and stacktrace_ops/print_trace_ops are used.

In the case where a thread is dumping its own thread, dump_stack() is called, and still works, I believe.

[Peter Jones]

Yang Sheng

Could you please investigate?

Thanks

Peter

[Andreas Dilger]

Whomever is working on this, please also update the error message to print kernel doesn't export dump_trace() since show_stack() hasn't existed for a long time.

It looks like the relevant kernel commit is:

commit e18bcccd1a4ecb41e99678e002ef833586185bf1
Author:     Josh Poimboeuf <[e-mail]>
AuthorDate: Fri Sep 16 14:18:16 2016 -0500
Commit:     Ingo Molnar <[e-mail]>
CommitDate: Tue Sep 20 08:29:34 2016 +0200

    x86/dumpstack: Convert show_trace_log_lvl() to use the new unwinder
    
    Convert show_trace_log_lvl() to use the new unwinder.  dump_trace() has
    been deprecated.

[Andreas Dilger]

This was disabled in Lustre via:

commit 70d70c4f541c84adc88c372d670cd3a7fa8bc91f
Author:     Dmitry Eremin <[e-mail]>

    LU-9183 libcfs: handle dump_trace() and related callbacks removal
    
    In kernel version 4.8 commit c8fe4609827aedc9c4b45de80e7cdc8ccfa8541b all previous
    users of dump_trace() have been converted to use the new unwind interfaces,
    so the dump_trace() and the related print_context_stack() and
    print_context_stack_bp() callback functions were removed.
    
    Change-Id: Ifa7a112d622b23f733d6daab05f9838afdf31a86
    Reviewed-on: https://review.whamcloud.com/25816

It seems we need to update to the new kernel API, since this was backported to the RHEL7.5 kernel.

[James A Simmons]

Maybe its time to look at using the hang checker available in the kernel?

[Andreas Dilger]

James, do you have any pointers to the kernel mechanism? We already get NMI watchdogs from the kernel, but those are when the thread doesn't schedule for a long time. Our current code can dump the stack on another service thread that is not making progress, even though it isn't totally dead (e.g. in a sleep/check loop). The thread can "ping" the watchdog to tell it is is alive, and lack of pings == lack of progress. If the kernel can do something similar, especially if it's been around a while, then I'd be happy to see it.

[Oleg Drokin]

hangcheck timer reports on idle threads that did not schedule in a while or some such (it needs to be compiled in and enabled), but it's not quite what we need since our timeouts are different and we have a bit different model of when to start the countdown and when to stop it.

[James A Simmons]

I was talking to Neil and he recommend we use save_stack_trace_tsk() since it been around since 2.6.29.

Also of note Shadow pointed out that the kernel does have a soft lockup thread (CONFIG_LOCKUP_DETECTOR). It appears enabled for RHEL kernels. Their is also a detect hung_task functionality.

[SC Admin (Inactive)]

FWIW we got hung tasks in LU-11082 and got this message and no stack traces. in retrospect I guess I should have tried sysrq to get stack traces before rebooting. I've got used to Lustre doing it for me.

cheers,
robin

[Gerrit Updater]

Yang Sheng ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32952
Subject: LU-11062 libcfs: use save_stack_trace for stack dump
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bc344d10030aa60385f0577a855993733b38c916

[Yang Sheng]

I have submitted a patch for this ticket. Just use save_stack_trace_tsk for backtrace dump. The obvious problem is that unable to judge a address whether reliable or not. This will make things confused in some case.

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32972
Subject: LU-11062 libcfs: use save_stack_trace for stack dump
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 7232c445fe30f6500f6f731ef8ffad617490eb68

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32952/
Subject: LU-11062 libcfs: use save_stack_trace for stack dump
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: afedf9343686504c89f2e28cf6133540166f2347

[Peter Jones]

Landed for 2.12

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/32972/
Subject: LU-11062 libcfs: use save_stack_trace for stack dump
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: a2af371bd8a79e293a9ba95b8016de92040101a6