[LU-10691] sanity-sec test_22: client hit Kernel panic - not syncing: softlockup: hung tasks

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Minor
• Affects Version/s: Lustre 2.11.0
• Environment: lustre-master tag-2.10.58 RHEL7.4 ZFS
• Severity: 3
• Assignee: WC Triage
• Reporter: Sarah Liu

Description

https://testing.hpdd.intel.com/test_sets/88c8f21a-111f-11e8-a7cd-52540065bddc

client 2 console

[19453.056075] Lustre: DEBUG MARKER: chmod 040 /mnt/lustre/d22.sanity-sec
[19453.959547] Lustre: DEBUG MARKER: chmod 050 /mnt/lustre/d22.sanity-sec
[19480.233808] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [khugepaged:33]
[19480.233808] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core iosf_mbi crc32_pclmul ghash_clmulni_intel nfsd ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev auth_rpcgss nfs_acl virtio_balloon parport_pc lockd parport grace i2c_piix4 sunrpc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect virtio_blk sysimgblt fb_sys_fops ttm 8139too crct10dif_pclmul crct10dif_common ata_piix crc32c_intel drm serio_raw virtio_pci 8139cp virtio_ring virtio mii i2c_core libata floppy
[19480.233808] CPU: 1 PID: 33 Comm: khugepaged Tainted: G           OE  ------------   3.10.0-693.17.1.el7.x86_64 #1
[19480.233808] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[19480.233808] task: ffff88007b85cf10 ti: ffff8800797a4000 task.ti: ffff8800797a4000
[19480.233808] RIP: 0010:[<ffffffff813323c2>]  [<ffffffff813323c2>] clear_page+0x12/0x40
[19480.233808] RSP: 0018:ffff8800797a7d80  EFLAGS: 00010216
[19480.233808] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000003f
[19480.233808] RDX: 000000000001bef8 RSI: 0000000000000008 RDI: ffff880000523000
[19480.233808] RBP: ffff8800797a7e38 R08: 0000000000000014 R09: a800062dcd000000
[19480.233808] R10: 57ffe3d2348b7340 R11: 0000000000000000 R12: ffffffff816b9bff
[19480.233808] R13: ffffffff816b9c06 R14: ffffffff816b9c0d R15: ffffffff816b9c14
[19480.233808] FS:  0000000000000000(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
[19480.233808] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[19480.233808] CR2: 00007fd3aa074000 CR3: 00000000019fa000 CR4: 00000000000606e0
[19480.233808] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[19480.233808] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[19480.233808] Call Trace:
[19480.233808]  [<ffffffff811ed21d>] ? khugepaged_scan_mm_slot+0x84d/0xcf0
[19480.233808]  [<ffffffff811ed7fb>] khugepaged+0x13b/0x480
[19480.233808]  [<ffffffff810b3690>] ? wake_up_atomic_t+0x30/0x30
[19523.626006] Lustre: DEBUG MARKER: chmod 060 /mnt/lustre/d22.sanity-sec
[19480.233808]  [<ffffffff811ed6c0>] ? khugepaged_scan_mm_slot+0xcf0/0xcf0
[19480.233808]  [<ffffffff810b270f>] kthread+0xcf/0xe0
[19480.233808]  [<ffffffff810b2640>] ? insert_kthread_work+0x40/0x40
[19480.233808]  [<ffffffff816b8798>] ret_from_fork+0x58/0x90
[19480.233808]  [<ffffffff810b2640>] ? insert_kthread_work+0x40/0x40
[19480.233808] Code: f3 48 ab c3 0f 1f 44 00 00 b9 00 10 00 00 31 c0 f3 aa c3 66 0f 1f 44 00 00 31 c0 b9 40 00 00 00 66 0f 1f 84 00 00 00 00 00 ff c9 <48> 89 07 48 89 47 08 48 89 47 10 48 89 47 18 48 89 47 20 48 89 
[19480.233808] Kernel panic - not syncing: softlockup: hung tasks
[19480.233808] CPU: 1 PID: 33 Comm: khugepaged Tainted: G           OEL ------------   3.10.0-693.17.1.el7.x86_64 #1
[19480.233808] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[19480.233808] Call Trace:
[19480.233808]  <IRQ>  [<ffffffff816a6071>] dump_stack+0x19/0x1b
[19480.233808]  [<ffffffff8169ff34>] panic+0xe8/0x20d
[19480.233808]  [<ffffffff8102d76f>] ? show_regs+0x5f/0x210
[19480.233808]  [<ffffffff81131531>] watchdog_timer_fn+0x221/0x230
[19480.233808]  [<ffffffff81131310>] ? watchdog+0x40/0x40
[19480.233808]  [<ffffffff810b6864>] __hrtimer_run_queues+0xd4/0x260
[19480.233808]  [<ffffffff810b6dff>] hrtimer_interrupt+0xaf/0x1d0
[19480.233808]  [<ffffffff81053a05>] local_apic_timer_interrupt+0x35/0x60
[19480.233808]  [<ffffffff816bea4d>] smp_apic_timer_interrupt+0x3d/0x50
[19480.233808]  [<ffffffff816b9d32>] apic_timer_interrupt+0x232/0x240
[19480.233808]  <EOI>  [<ffffffff813323c2>] ? clear_page+0x12/0x40
[19480.233808]  [<ffffffff811ed21d>] ? khugepaged_scan_mm_slot+0x84d/0xcf0
[19480.233808]  [<ffffffff811ed7fb>] khugepaged+0x13b/0x480
[19480.233808]  [<ffffffff810b3690>] ? wake_up_atomic_t+0x30/0x30
[19480.233808]  [<ffffffff811ed6c0>] ? khugepaged_scan_mm_slot+0xcf0/0xcf0
[19480.233808]  [<ffffffff810b270f>] kthread+0xcf/0xe0
[19480.233808]  [<ffffffff810b2640>] ? insert_kthread_work+0x40/0x40
[19480.233808]  [<ffffffff816b8798>] ret_from_fork+0x58/0x90
[19480.233808]  [<ffffffff810b2640>] ? insert_kthread_work+0x40/0x40
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-693.17.1.el7.x86_64 ([e-mail]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Thu Jan 25 20:13:58 UTC 2018
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.17.1.el7.x86_64 root=UUID=36a4fa8e-8395-4c4c-9d40-93a0779cd2bb ro console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never disable_cpu_apicid=0 elfcorehdr=867708K
[    0.000000] Disabled fast string operations
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000000fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000001000-0x000000000009dbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009dc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000021000000-0x0000000034f5efff] usable
[    0.000000] BIOS-e820: [mem 0x0000000034fffc00-0x0000000034ffffff] usable
[    0.000000] BIOS-e820: [mem 0x000000007fffd000-0x000000007fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffbc000-0x00000000ffffffff] reserved
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.4 present.
[    0.000000] Hypervisor detected: KVM