[LU-1587] watchdog detected hard LOCKUP on OSS under heavy client reconnects Created: 30/Jun/12 Updated: 13/Apr/14 Resolved: 13/Apr/14 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.1.1 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Blocker |
| Reporter: | Ned Bass | Assignee: | Niu Yawei (Inactive) |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Environment: | |||
| Severity: | 3 |
| Rank (Obsolete): | 3981 |
| Description |
|
We are experiencing a severe disruption of service on our classified network. Beginning around 5pm yesterday (Friday Jun 29) we began to have many evictions and reconnections across multiple servers and across multiple lustre clusters on our classified network. Some sort of network event may have triggered this, but we're not sure at this point. Lustre servers started getting STONITH'd by their failover partners. Many servers have been getting panics due to CPU hard lockup. The stack traces aren't helpful; they just show the swapper task handling the watchdog NMI. We're trying to quiesce things by stopping lnet on the compute cluster routers, getting lustre started on all the servers, then bringing client clusters back online one at a time. However, we are still having sporadic CPU hard lockups on OSS's as clients reconnect. Typically the lockup follows several thousand client reconnects. We have some complete crash dumps to look at, but there's not much to go on. Some process must be leaving interrupts disabled, but I'm not sure how to go about identifying with the data we have. |
| Comments |
| Comment by Peter Jones [ 01/Jul/12 ] |
|
Niu could you please look into this one Peter |
| Comment by Niu Yawei (Inactive) [ 02/Jul/12 ] |
|
Could you post the stack trace here as well? Though it might not be helpful. Thanks. |
| Comment by Prakash Surya (Inactive) [ 02/Jul/12 ] |
|
Hi Niu. I'll post the stack trace as soon as I can. Although, I briefly talked to Ned and he didn't think the stack trace would be helpful. |
| Comment by Prakash Surya (Inactive) [ 02/Jul/12 ] |
|
Here's a stack trace from one of the nodes which crashed. Please excuse any typo's, I had to copy it by hand: Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 4 PID: 0 TASK: ... CPU: 4 COMMAND: "swapper" 0. machine_kexec 1. crash_kexec 2. panic 3. watchdog_overflow_callback 4. __perf_event_overflow 5. perf_event_overlfow 6. x86_pmu_handle_irq 7. perf_event_nmi_handler 8. notifier_call_chain 9. atomic_notifier_call_chain 10. notify_die 11. do_nmi 12. nmi 13. ktime_get 14. tick_check_idle 15. irq_enter 16. smp_apic_timer_interrupt 17. apic_timer_interrupt 18. apic_timer_interrupt 19. notifier_call_chain 20. cpu_idle |
| Comment by Prakash Surya (Inactive) [ 02/Jul/12 ] |
|
This curiously lines up with many 'leap second' issues I've read about over the weekend. Although, it's pure speculation if this is issue is related. |
| Comment by Andreas Dilger [ 13/Apr/14 ] |
|
Haven't seen any more information about this bug, and unable to make any progress based on the description here. Based on the later comments, this might have been caused by the leap second bug. |