[LU-11062] Backtrace stack printing is broken in RHEL 7.5

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.12.0, Lustre 2.10.5
• Severity: 3
• Assignee: Yang Sheng
• Reporter: Oleg Drokin

Description

It looks like struct stacktrace_ops is no longer there in rhel7.5 so our detection of if dump_stack would work is no longer working.

We get this now:

[29185.979935] LNet: Service thread pid 9373 was inactive for 40.04s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[29185.981691] LNet: 31237:0:(linux-debug.c:185:libcfs_call_trace()) can't show stack: kernel doesn't export show_task
[29185.982826] LustreError: dumping log to /tmp/lustre-log.1527605717.9373

Given that struct stacktrace_ops are still there in mainline kernels, this seems something specific to rhel7.5 and we need to find another way of detecting this I guess.

Issue Links

• is related to: LU-11241 Unable to umount during MDT fail back
• is related to: LU-9183 Support for linux 4.9 kernels

Comments

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/32972/
Subject: LU-11062 libcfs: use save_stack_trace for stack dump
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: a2af371bd8a79e293a9ba95b8016de92040101a6

[Peter Jones]

Landed for 2.12

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32952/
Subject: LU-11062 libcfs: use save_stack_trace for stack dump
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: afedf9343686504c89f2e28cf6133540166f2347

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32972
Subject: LU-11062 libcfs: use save_stack_trace for stack dump
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 7232c445fe30f6500f6f731ef8ffad617490eb68

[Yang Sheng]

I have submitted a patch for this ticket. Just use save_stack_trace_tsk for backtrace dump. The obvious problem is that unable to judge a address whether reliable or not. This will make things confused in some case.

[Gerrit Updater]

Yang Sheng ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32952
Subject: LU-11062 libcfs: use save_stack_trace for stack dump
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bc344d10030aa60385f0577a855993733b38c916

[SC Admin]

FWIW we got hung tasks in LU-11082 and got this message and no stack traces. in retrospect I guess I should have tried sysrq to get stack traces before rebooting. I've got used to Lustre doing it for me.

cheers,
robin