[LU-9002] obdfilter-survey test_3a:Kernel panic - not syncing: softlockup: hung tasks

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.10.0
• Environment: Full - RHEL7.3 Server/Client DNE; Master, build# 3486
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for Saurabh Tandan <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/207d436c-d4d2-11e6-8b52-5254006e85c2.

The sub-test test_3a failed with the following error:

test failed to respond and timed out

MDS Console:

00:54:49:[47828.073657] Code: 45 c0 c7 45 c8 00 00 00 00 8b 40 0c 89 45 a4 66 0f 1f 44 00 00 49 8b 47 18 48 8d 75 c0 4c 89 ff ff 10 48 85 c0 0f 84 8a 01 00 00 <48> 8b 18 48 85 db 0f 84 66 01 00 00 49 8b 47 08 48 89 de 4c 89 
00:54:49:[47828.073657] Kernel panic - not syncing: softlockup: hung tasks
00:54:49:[47828.073657] CPU: 1 PID: 5449 Comm: umount Tainted: G           OEL ------------   3.10.0-514.2.2.el7_lustre.x86_64 #1
00:54:49:[47828.073657] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
00:54:49:[47828.073657]  ffffffff818d9745 0000000087194aeb ffff88007fd03e18 ffffffff81686318
00:54:49:[47828.073657]  ffff88007fd03e98 ffffffff8167f743 0000000000000008 ffff88007fd03ea8
00:54:49:[47828.073657]  ffff88007fd03e48 0000000087194aeb ffff88007fd03e67 0000000000000000
00:54:49:[47828.073657] Call Trace:
00:54:49:[47828.073657]  <IRQ>  [<ffffffff81686318>] dump_stack+0x19/0x1b
00:54:49:[47828.073657]  [<ffffffff8167f743>] panic+0xe3/0x1f2
00:54:49:[47828.073657]  [<ffffffff8112eedc>] watchdog_timer_fn+0x20c/0x220
00:54:49:[47828.073657]  [<ffffffff8112ecd0>] ? watchdog+0x50/0x50
00:54:49:[47828.073657]  [<ffffffff810b4982>] __hrtimer_run_queues+0xd2/0x260
00:54:49:[47828.073657]  [<ffffffff810b4f20>] hrtimer_interrupt+0xb0/0x1e0
00:54:49:[47828.073657]  [<ffffffff816983dc>] ? call_softirq+0x1c/0x30
00:54:49:[47828.073657]  [<ffffffffa0a3cd50>] ? cleanup_resource+0x370/0x370 [ptlrpc]
00:54:49:[47828.073657]  [<ffffffff81051087>] local_apic_timer_interrupt+0x37/0x60
00:54:49:[47828.073657]  [<ffffffff8169904f>] smp_apic_timer_interrupt+0x3f/0x60
00:54:49:[47828.073657]  [<ffffffff8169759d>] apic_timer_interrupt+0x6d/0x80
00:54:49:[47828.073657]  <EOI>  [<ffffffffa073440e>] ? cfs_hash_for_each_relax+0x16e/0x400 [libcfs]
00:54:49:[47828.073657]  [<ffffffffa0734405>] ? cfs_hash_for_each_relax+0x165/0x400 [libcfs]
00:54:49:[47828.073657]  [<ffffffffa0a3cd50>] ? cleanup_resource+0x370/0x370 [ptlrpc]
00:54:49:[47828.073657]  [<ffffffffa0a3cd50>] ? cleanup_resource+0x370/0x370 [ptlrpc]
00:54:49:[47828.073657]  [<ffffffffa0737645>] cfs_hash_for_each_nolock+0x75/0x1c0 [libcfs]
00:54:49:[47828.073657]  [<ffffffffa0a3ae70>] ldlm_namespace_cleanup+0x30/0xc0 [ptlrpc]
00:54:49:[47828.073657]  [<ffffffffa0a3bd1f>] __ldlm_namespace_free+0x5f/0x5c0 [ptlrpc]
00:54:49:[47828.073657]  [<ffffffff811dcc63>] ? kfree+0x103/0x140
00:54:49:[47828.073657]  [<ffffffffa085f971>] ? keys_fini+0xb1/0x1d0 [obdclass]
00:54:49:[47828.073657]  [<ffffffff811dcc63>] ? kfree+0x103/0x140
00:54:49:[47828.073657]  [<ffffffffa085f971>] ? keys_fini+0xb1/0x1d0 [obdclass]
00:54:49:[47828.073657]  [<ffffffffa0a3c2da>] ldlm_namespace_free_prior+0x5a/0x210 [ptlrpc]
00:54:49:[47828.073657]  [<ffffffffa0e44920>] mdt_device_fini+0x300/0x10f0 [mdt]
00:54:49:[47828.073657]  [<ffffffffa084c5fc>] class_cleanup+0x8ec/0xd80 [obdclass]
00:54:49:[47828.073657]  [<ffffffffa084eff4>] class_process_config+0x1e44/0x2f80 [obdclass]
00:54:49:[47828.073657]  [<ffffffffa083e5a9>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
00:54:49:[47828.073657]  [<ffffffffa085021f>] class_manual_cleanup+0xef/0x810 [obdclass]
00:54:49:[47828.073657]  [<ffffffffa087e6de>] server_put_super+0x8de/0xcd0 [obdclass]
00:54:49:[47828.073657]  [<ffffffff81200802>] generic_shutdown_super+0x72/0xf0
00:54:49:[47828.073657]  [<ffffffff81200bd2>] kill_anon_super+0x12/0x20
00:54:49:[47828.073657]  [<ffffffffa0853982>] lustre_kill_super+0x32/0x50 [obdclass]
00:54:49:[47828.073657]  [<ffffffff81200f89>] deactivate_locked_super+0x49/0x60
00:54:49:[47828.073657]  [<ffffffff81201586>] deactivate_super+0x46/0x60
00:54:49:[47828.073657]  [<ffffffff8121e9c5>] mntput_no_expire+0xc5/0x120
00:54:49:[47828.073657]  [<ffffffff8121fb00>] SyS_umount+0xa0/0x3b0
00:54:49:[47828.073657]  [<ffffffff81696949>] system_call_fastpath+0x16/0x1b
00:54:49:[    0.000000] Initializing cgroup subsys cpuset
00:54:49:[    0.000000] Initializing cgroup subsys cpu
00:54:49:[    0.000000] Initializing cgroup subsys cpuacct
00:54:49:[    0.000000] Linux version 3.10.0-514.2.2.el7_lustre.x86_64 ([e-mail]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Dec 27 15:41:56 PST 2016
00:54:49:[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-514.2.2.el7_lustre.x86_64 root=UUID=4fbb23c5-e2f5-4e65-bc16-0d004c982271 ro console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never disable_cpu_apicid=0 elfcorehdr=851324K
00:54:49:[    0.000000] Disabled fast string operations
00:54:49:[    0.000000] e820: BIOS-provided physical RAM map:
00:54:49:[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000000fff] reserved
00:54:49:[    0.000000] BIOS-e820: [mem 0x0000000000001000-0x000000000009dbff] usable
00:54:49:[    0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved
00:54:49:[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
00:54:49:[    0.000000] BIOS-e820: [mem 0x0000000020000000-0x0000000033f5efff] usable
00:54:49:[    0.000000] BIOS-e820: [mem 0x0000000033fffc00-0x0000000033ffffff] usable
00:54:49:[    0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved
00:54:49:[    0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved
00:54:49:[    0.000000] NX (Execute Disable) protection: active
00:54:49:[    0.000000] SMBIOS 2.4 present.
00:54:49:[    0.000000] Hypervisor detected: KVM
00:54:49:[    0.000000] e820: last_pfn = 0x34000 max_arch_pfn = 0x400000000
00:54:49:[    0.000000] x86 PAT enabled: cpu 0, old 0x70106, new 0x7010600070106
00:54:49:[    0.000000] x2apic enabled by BIOS, switching to x2apic ops
00:54:49:[    0.000000] found SMP MP-table at [mem 0x000fda30-0x000fda3f] mapped at [ffff8800000fda30]
00:54:49:[    0.000000] iBFT found at 0x9aff0.
00:54:49:[    0.000000] Using GB pages for direct mapping
00:54:49:[    0.000000] RAMDISK: [mem 0x30a32000-0x31ffffff]
00:54:49:[    0.000000] ACPI: RSDP 00000000000fd9e0 00014 (v00 BOCHS )
00:54:49:[    0.000000] ACPI: RSDT 000000007fffd5d0 00034 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
00:54:49:[    0.000000] ACPI: FACP 000000007ffffe20 00074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
00:54:49:[    0.000000] ACPI: DSDT 000000007fffd910 024A2 (v01   BXPC   BXDSDT 00000001 INTL 20090123)
00:54:49:[    0.000000] ACPI: FACS 000000007ffffdc0 00040
00:54:49:[    0.000000] ACPI: SSDT 000000007fffd810 000FF (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
00:54:49:[    0.000000] ACPI: APIC 000000007fffd720 00080 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
00:54:49:[    0.000000] ACPI: SSDT 000000007fffd610 0010F (v01   BXPC BXSSDTPC 00000001 INTL 20090123)
00:54:49:[    0.000000] Setting APIC routing to cluster x2apic.