[LU-9705] sanity-benchmark test_iozone: client hit Kernel panic - not syncing: softlockup: hung tasks

Mon, 9 Oct 2017 22:18:57 +0000

This issue was created by maloo for sarah_lw <wei3.liu@intel.com>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/caed7928-52c9-11e7-bacf-5254006e85c2.

The sub-test test_iozone failed with the following error:

test failed to respond and timed out

server: tag 2.9.59.15 #3603
client: SLES12SP2

client 2 console

20:18:17:[ 3648.554839] Lustre: DEBUG MARKER: == sanity-benchmark test iozone: iozone ============================================================== 10:18:03 (1497547083)
20:18:17:[ 3650.500317] Lustre: DEBUG MARKER: /usr/sbin/lctl mark min OST has 1782832kB available, using 3833384kB file size
20:18:17:[ 3650.799799] Lustre: DEBUG MARKER: min OST has 1782832kB available, using 3833384kB file size
20:18:17:[14286.666391] Lustre: 8737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497557703/real 1497557703]  req@ffff88007910a640 x1570288350647040/t0(0) o400->lustre-MDT0000-mdc-ffff880036ecc800@10.9.4.27@tcp:12/10 lens 224/224 e 0 to 1 dl 1497557710 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
20:18:17:[14293.242874] Lustre: lustre-MDT0000-mdc-ffff880036ecc800: Connection to lustre-MDT0000 (at 10.9.4.27@tcp) was lost; in progress operations using this service will wait for recovery to complete
20:18:17:[14295.985268] Lustre: 8737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1497557703/real 1497557703]  req@ffff88007910a040 x1570288350647072/t0(0) o400->lustre-OST0001-osc-ffff880036ecc800@10.9.4.22@tcp:28/4 lens 224/224 e 0 to 1 dl 1497557710 ref 1 fl Rpc:RXN/0/ffffffff rc 0/-1
20:18:17:[14298.110548] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [ptlrpcd_00_00:8737]
20:18:17:[14298.110548] Modules linked in: osc(OEN) mgc(OEN) lustre(OEN) lmv(OEN) fld(OEN) mdc(OEN) fid(OEN) lov(OEN) ksocklnd(OEN) ptlrpc(OEN) obdclass(OEN) lnet(OEN) libcfs(OEN) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache af_packet iscsi_boot_sysfs ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm configfs ib_cm iw_cm ib_sa ib_mad
20:18:17:[14338.508455] INFO: rcu_sched detected stalls on CPUs/tasks:
20:18:17:[14338.508455] 	1-...: (8 ticks this GP) idle=225/140000000000001/0 softirq=505433/505433 fqs=2146 
20:18:17:[14338.508455] 	(detected by 0, t=15224 jiffies, g=105017, c=105016, q=1386)
20:18:17:[14338.508455] Task dump for CPU 1:
20:18:17:[14338.508455] ptlrpcd_00_00   R  running task        0  8737      2 0x00000008
20:18:17:[14338.508455]  000000000000ffff 0000000000000537 ffff8800369a4800 0000000000000000
20:18:17:[14338.508455]  0000000000808001 ffff88007ab0e000 ffff88007a489008 ffffffffffffff10
20:18:17:[14338.508455]  ffffffffa0de98f0 0000000000000010 0000000000010292 ffff88007bf3bd38
20:18:17:[14338.508455] Call Trace:
20:18:17:[14327.366836]  ib_core ib_addrInexact backtrace:
20:18:17:[14338.508455] 
20:18:17:[14327.366836]  crct10dif_pclmul [<ffffffffa0de98f0>] ? brw_interpret+0xdd0/0xdd0 [osc]
20:18:17:[14327.366836]  crc32_pclmul [<ffffffffa0a7e905>] ? ptlrpc_set_import_discon+0x2d5/0x540 [ptlrpc]
20:18:17:[14327.366836]  crc32c_intel [<ffffffffa0a7eba5>] ? ptlrpc_fail_import+0x35/0x1f0 [ptlrpc]
20:18:17:[14327.366836]  ghash_clmulni_intel [<ffffffffa0a57c09>] ? ptlrpc_expire_one_request+0x349/0x420 [ptlrpc]
20:18:17:[14327.366836]  jitterentropy_rng [<ffffffffa0a57d77>] ? ptlrpc_expired_set+0x97/0x170 [ptlrpc]
20:18:17:[14327.366836]  drbg [<ffffffffa0a8508e>] ? ptlrpcd+0x22e/0x490 [ptlrpc]
20:18:17:[14338.508455]  [<ffffffff810a5fa0>] ? wake_up_q+0x80/0x80
20:18:17:[14327.366836]  ansi_cprng [<ffffffffa0a84e60>] ? ptlrpcd_check+0x570/0x570 [ptlrpc]
20:18:17:[14327.366836]  aesni_intel [<ffffffff8109999d>] ? kthread+0xbd/0xe0
20:18:17:[14338.508455]  [<ffffffff810998e0>] ? kthread_park+0x50/0x50
20:18:17:[14338.508455]  [<ffffffff815e437f>] ? ret_from_fork+0x3f/0x70
20:18:17:[14327.366836]  8139too [<ffffffff810998e0>] ? kthread_park+0x50/0x50
20:18:17:[14338.508455] rcu_sched kthread starved for 3220 jiffies! g105017 c105016 f0x0 s3 ->state=0x0
20:18:17:[14327.366836]  aes_x86_64 lrw gf128mul[14403.677716] Lustre: lustre-MDT0000-mdc-ffff880036ecc800: Connection restored to 10.9.4.27@tcp (at 10.9.4.27@tcp)
20:18:17:[14403.678530]  glue_helper ablk_helper 8139cp cryptd ppdev joydev pcspkr virtio_balloon mii i2c_piix4 parport_pc parport pvpanic acpi_cpufreq processor button ata_generic ext4 crc16 jbd2 mbcache floppy virtio_blk ata_piix ahci libahci cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw uhci_hcd ehci_hcd libata usbcore usb_common virtio_pci virtio_ring virtio drm sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_mod autofs4
20:18:17:[14403.741812] Supported: No, Unsupported modules are loaded
20:18:17:[14403.741812] CPU: 1 PID: 8737 Comm: ptlrpcd_00_00 Tainted: G           OE   N  4.4.59-92.17-default #1
20:18:17:[14403.741812] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
20:18:17:[14403.741812] task: ffff88007c02d080 ti: ffff88007bf38000 task.ti: ffff88007bf38000
20:18:17:[14403.741812] RIP: 0010:[<ffffffffa0de98f0>]  [<ffffffffa0de98f0>] osc_import_event+0x0/0x1250 [osc]
20:18:17:[14403.741812] RSP: 0018:ffff88007bf3bd38  EFLAGS: 00010292
20:18:17:[14403.741812] RAX: ffff8800369a4800 RBX: ffff88007ab0e000 RCX: 0000000000000000
20:18:17:[14403.741812] RDX: 0000000000808001 RSI: ffff88007ab0e000 RDI: ffff88007a489008
20:18:17:[14403.741812] RBP: ffff88007a489008 R08: 0000000000000537 R09: 000000000000ffff
20:18:17:[14403.741812] R10: 6174732074726f70 R11: ffff880067d24ac9 R12: 000200000a090416
20:18:17:[14403.741812] R13: 0000000000000001 R14: ffff880036d13280 R15: ffff88007c02d080
20:18:17:[14403.741812] FS:  0000000000000000(0000) GS:ffff88007fd00000(0000) knlGS:0000000000000000
20:18:17:[14403.741812] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
20:18:17:[14403.741812] CR2: 00000000022c14b8 CR3: 0000000000063000 CR4: 00000000000406e0
20:18:17:[14403.741812] Stack:
20:18:17:[14403.741812]  ffffffffa0a7e905 ffff88007c02d080 0000000000000016 0000000e00000000
20:18:17:[14403.741812]  ffff88007a489508 ffff88007ab0e000 0000000000000001 0000000000000001
20:18:17:[14403.741812]  ffffffffa0a7eba5 ffff88007910a040 ffff88007ab0e000 ffffffffa0a57c09
20:18:17:[14403.741812] Call Trace:
20:18:17:[14444.239498]  [<ffffffffa0a7e905>] ptlrpc_set_import_discon+0x2d5/0x540 [ptlrpc]
20:22:40:[14453.685607]  [<ffffffffa0a7eba5>] ptlrpc_fail_import+0x35/0x1f0 [ptlrpc]
20:22:40:[14453.842533]  [<ffffffffa0a57c09>] ptlrpc_expire_one_request+0x349/0x420 [ptlrpc]
20:22:40:[14453.842533]  [<ffffffffa0a57d77>] ptlrpc_expired_set+0x97/0x170 [ptlrpc]
20:22:40:[14453.842533]  [<ffffffffa0a8508e>] ptlrpcd+0x22e/0x490 [ptlrpc]
20:22:40:[14453.842533]  [<ffffffff8109999d>] kthread+0xbd/0xe0
20:22:40:[14453.842533]  [<ffffffff815e437f>] ret_from_fork+0x3f/0x70
20:22:40:[14454.079581] Lustre: lustre-OST0001-osc-ffff880036ecc800: Connection restored to 10.9.4.22@tcp (at 10.9.4.22@tcp)
20:22:40:[14461.156046] Lustre: 8738:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1497557889/real 0]  req@ffff88007a42ccc0 x1570288350647312/t0(0) o400->MGC10.9.4.27@tcp@10.9.4.27@tcp:26/25 lens 224/224 e 0 to 1 dl 1497557896 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
20:22:40:[14461.156066] LustreError: 166-1: MGC10.9.4.27@tcp: Connection to MGS (at 10.9.4.27@tcp) was lost; in progress operations using this service will fail
20:22:40:[14468.122634] Lustre: 8738:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1497557889/real 0]  req@ffff88007937f980 x1570288350647376/t0(0) o400->lustre-OST0002-osc-ffff880036ecc800@10.9.4.22@tcp:28/4 lens 224/224 e 0 to 1 dl 1497557896 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
20:22:40:[14468.122653] Lustre: lustre-OST0002-osc-ffff880036ecc800: Connection to lustre-OST0002 (at 10.9.4.22@tcp) was lost; in progress operations using this service will wait for recovery to complete
20:22:40:[14468.122662] Lustre: Skipped 1 previous similar message
20:22:40:[14473.709403] Lustre: 8736:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1497557903/real 0]  req@ffff88007c1d5340 x1570288350648208/t0(0) o250->MGC10.9.4.27@tcp@10.9.4.27@tcp:26/25 lens 520/544 e 0 to 1 dl 1497557909 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
20:22:40:[14473.709407] Lustre: 8736:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 51 previous similar messages
20:22:40:[14508.004015] DWARF2 unwinder stuck at ret_from_fork+0x3f/0x70
20:22:40:[14508.004015] 
20:22:40:[14508.004015] Leftover inexact backtrace:
20:22:40:[14508.004015] 
20:22:40:[14508.004015]  [<ffffffff810998e0>] ? kthread_park+0x50/0x50
20:22:40:[14508.004015] Code: c7 05 89 89 03 00 00 00 04 00 e8 ac 36 8e ff 48 c7 c7 40 22 e2 a0 e8 20 8e 8d ff e8 1a 35 02 00 90 66 2e 0f 1f 84 00 00 00 00 00 <66> 66 66 66 90 41 54 49 89 fc 55 48 89 f5 53 48 83 ec 18 f6 05 
20:22:40:[14508.004015] Kernel panic - not syncing: softlockup: hung tasks
20:22:40:[14508.272024] CPU: 1 PID: 8737 Comm: ptlrpcd_00_00 Tainted: G           OEL  N  4.4.59-92.17-default #1
20:22:40:[14508.272024] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
20:22:40:[14508.272024]  0000000000000000 ffffffff8130f110 ffffffff81860d69 ffff88007fd03ed0
20:22:40:[14508.272024]  ffffffff81182771 0000000000000008 ffff88007fd03ee0 ffff88007fd03e80
20:22:40:[14508.272024]  ffffffff82008980 0000000000000000 0000000000000001 0000000000000000
20:22:40:[14508.272024] Call Trace:
20:22:40:[14508.272024]  [<ffffffff81019a99>] dump_trace+0x59/0x310
20:22:40:[14508.272024]  [<ffffffff81019e3a>] show_stack_log_lvl+0xea/0x170
20:22:40:[14508.272024]  [<ffffffff8101abc1>] show_stack+0x21/0x40
20:22:40:[14508.272024]  [<ffffffff8130f110>] dump_stack+0x5c/0x7c
20:22:40:[14508.272024]  [<ffffffff81182771>] panic+0xd2/0x219
20:22:40:[14508.272024]  [<ffffffff81136359>] watchdog_timer_fn+0x1d9/0x1e0
20:22:40:[14508.272024]  [<ffffffff810eadaa>] __hrtimer_run_queues+0xea/0x260
20:22:40:[14508.272024]  [<ffffffff810eb1e9>] hrtimer_interrupt+0x99/0x190
20:22:40:[14508.272024]  [<ffffffff815e6ce9>] smp_apic_timer_interrupt+0x39/0x50
20:22:40:[14508.272024]  [<ffffffff815e4dbc>] apic_timer_interrupt+0x8c/0xa0
20:22:40:[14508.272024] DWARF2 unwinder stuck at apic_timer_interrupt+0x8c/0xa0
20:22:40:[14508.272024] 
20:22:40:[14508.272024] Leftover inexact backtrace:
20:22:40:[14508.272024] 
20:22:40:[14508.272024]  <IRQ>  <EOI>  [<ffffffffa0de98f0>] ? brw_interpret+0xdd0/0xdd0 [osc]
20:22:40:[14508.272024]  [<ffffffffa0a7e905>] ? ptlrpc_set_import_discon+0x2d5/0x540 [ptlrpc]
20:22:40:[14508.272024]  [<ffffffffa0a7eba5>] ? ptlrpc_fail_import+0x35/0x1f0 [ptlrpc]
20:22:40:[14508.272024]  [<ffffffffa0a57c09>] ? ptlrpc_expire_one_request+0x349/0x420 [ptlrpc]
20:22:40:[14508.272024]  [<ffffffffa0a57d77>] ? ptlrpc_expired_set+0x97/0x170 [ptlrpc]
20:22:40:[14508.272024]  [<ffffffffa0a8508e>] ? ptlrpcd+0x22e/0x490 [ptlrpc]
20:22:40:[14508.272024]  [<ffffffff810a5fa0>] ? wake_up_q+0x80/0x80
20:22:40:[14508.272024]  [<ffffffffa0a84e60>] ? ptlrpcd_check+0x570/0x570 [ptlrpc]
20:22:40:[14508.272024]  [<ffffffff8109999d>] ? kthread+0xbd/0xe0
20:22:40:[14508.272024]  [<ffffffff810998e0>] ? kthread_park+0x50/0x50
20:22:40:[14508.272024]  [<ffffffff815e437f>] ? ret_from_fork+0x3f/0x70
20:22:40:[14508.272024]  [<ffffffff810998e0>] ? kthread_park+0x50/0x50
20:22:40:[    0.004000] do_IRQ: 0.97 No irq handler for vector
20:22:40:[    0.090545] ioremap error for 0x7ffff000-0x80000000, requested 0x2, got 0x0
20:22:40:[    0.091197] dmi: Firmware registration failed.
20:22:40:[    9.058417] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver
20:22:40:[   10.697136] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
20:22:40:[   10.697176] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver
20:22:40:[�[0m�[0;31m*     �[0m] (1 of 2) A start job is running for...om the Real Root (6s / no limit)[   15.278786] [drm] Initialized drm 1.1.0 20060810
20:22:40:[   15.296022] ACPI: bus type USB registered
20:22:40:[   15.296045] usbcore: registered new interface driver usbfs
20:22:40:[   15.296055] usbcore: registered new interface driver hub
20:22:40:[   15.296067] usbcore: registered new device driver usb
20:22:40:[   15.299239] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
20:22:40:[   15.330262] uhci_hcd: USB Universal Host Controller Interface driver
20:22:40:[   15.330521] uhci_hcd 0000:00:01.2: UHCI Host Controller
20:22:40:[   15.330526] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
20:22:40:[   15.330538] uhci_hcd 0000:00:01.2: detected 2 ports
20:22:40:[   15.389377] random: nonblocking pool is initialized
20:22:40:[   15.628576] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020
20:22:40:[   23.670257] FDC 0 is a S82078B
20:22:40:[   23.670367] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
20:22:40:[   23.670369] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
20:22:40:[   23.670370] usb usb1: Product: UHCI Host Controller
20:22:40:[   23.670371] usb usb1: Manufacturer: Linux 4.4.59-92.17-default uhci_hcd
20:22:40:[   23.670371] usb usb1: SerialNumber: 0000:00:01.2
20:22:40:[   23.670545] hub 1-0:1.0: USB hub found
20:22:40:[   23.670550] hub 1-0:1.0: 2 ports detected
20:22:40:[   23.676031] [TTM] Zone  kernel: Available graphics memory: 54438 kiB
20:22:40:[   23.676031] [TTM] Initializing pool allocator
20:22:40:[   23.676035] [TTM] Initializing DMA pool allocator
20:22:40:[   23.676285] [drm] fb mappable at 0xF0000000
20:22:40:[   23.676285] [drm] vram aper at 0xF0000000
20:22:40:[   23.676286] [drm] size 33554432
20:22:40:[   23.676286] [drm] fb depth is 24
20:22:40:[   23.676286] [drm]    pitch is 3072
20:22:40:[   37.422409] fbcon: cirrusdrmfb (fb0) is primary device
20:22:40:[   68.787302] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [systemd-udevd:187]
20:22:40:[   68.787381] Modules linked in: virtio_blk(+) ata_piix(+) cirrus(+) drm_kms_helper syscopyarea ahci libahci uhci_hcd sysfillrect sysimgblt fb_sys_fops ttm serio_raw floppy(+) ehci_hcd usbcore drm virtio_pci virtio_ring virtio usb_common libata sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_mod autofs4
20:22:40:[   68.787386] Supported: Yes
20:22:40:[   68.787392] CPU: 0 PID: 187 Comm: systemd-udevd Tainted: G        W          4.4.59-92.17-default #1
20:22:40:[   68.787395] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
20:22:40:[   68.787398] task: ffff8800305f4880 ti: ffff8800359f4000 task.ti: ffff8800359f4000
20:22:40:[   68.787429] RIP: 0010:[<ffffffff8131cb53>]  [<ffffffff8131cb53>] memset_orig+0x33/0xb0
20:22:40:[   68.787431] RSP: 0000:ffff8800359f72b8  EFLAGS: 00010212
20:22:40:[   68.787431] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000007dbf
20:22:40:[   68.787432] RDX: 0000000000240000 RSI: 0000000000000000 RDI: ffffc90000449000
20:22:40:[   68.787434] RBP: ffff8800302648d0 R08: 00000000000f1fff R09: 0000000000000000
20:22:40:[   68.787434] R10: ffffc90000400000 R11: 0000000000026084 R12: ffff880030241800
20:22:40:[   68.787437] R13: ffff8800359f7460 R14: 0000000000000000 R15: ffff8800302648b0
20:22:40:[   68.787438] FS:  00007fdf3331e8c0(0000) GS:ffff880033e00000(0000) knlGS:0000000000000000
20:22:40:[   68.787438] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
20:22:40:[   68.787439] CR2: 00007f08e4749000 CR3: 00000000359f1000 CR4: 00000000000406f0
20:22:40:[   68.787447] Stack:
20:22:40:[   68.787451]  ffffffffa01e148c 0000000000000000 0000000000000000 0000000000000000
20:22:40:[   68.787452]  0000000000000000 ffff88003023a840 ffff8800302648b0 ffff880030241838
20:22:40:[   68.787453]  0000000200000240 ffffffff00000000 0000000000000000 0000000000002000
20:22:40:[   68.787454] Call Trace:
20:22:40:[   68.789279] Inexact backtrace:
20:22:40:[   68.789279] 
20:22:40:[   68.789305]  [<ffffffffa01e148c>] ? ttm_bo_move_memcpy+0x5dc/0x690 [ttm]
20:22:40:[   68.789311]  [<ffffffffa01dec58>] ? ttm_bo_handle_move_mem+0x258/0x590 [ttm]
20:22:40:[   68.789313]  [<ffffffffa01df5d4>] ? ttm_bo_mem_space+0x134/0x390 [ttm]
20:22:40:[   68.789316]  [<ffffffffa01dfbde>] ? ttm_bo_validate+0x11e/0x130 [ttm]
20:22:40:[   68.789324]  [<ffffffffa026a5e6>] ? cirrus_bo_pin+0x76/0xa0 [cirrus]
20:22:40:[   68.789327]  [<ffffffffa0268c37>] ? cirrus_crtc_do_set_base.isra.9.constprop.11+0xe7/0x3a0 [cirrus]
20:22:40:[   68.789329]  [<ffffffffa026932e>] ? cirrus_crtc_mode_set+0x43e/0x4a0 [cirrus]
20:22:40:[   68.789350]  [<ffffffffa0232623>] ? drm_crtc_helper_set_mode+0x293/0x4b0 [drm_kms_helper]
20:22:40:[   68.789355]  [<ffffffffa023374d>] ? drm_crtc_helper_set_config+0x91d/0xb50 [drm_kms_helper]
20:22:40:[   68.789402]  [<ffffffffa01612c0>] ? drm_mode_set_config_internal+0x60/0xf0 [drm]
20:22:40:[   68.789407]  [<ffffffffa023ea4d>] ? restore_fbdev_mode+0xad/0x250 [drm_kms_helper]
20:22:40:[   68.789416]  [<ffffffffa016fe20>] ? drm_modeset_lock_all_ctx+0x90/0xa0 [drm]
20:22:40:[   68.789421]  [<ffffffffa0240bfe>] ? drm_fb_helper_restore_fbdev_mode_unlocked+0x2e/0x70 [drm_kms_helper]
20:22:40:[   68.789424]  [<ffffffffa0240c68>] ? drm_fb_helper_set_par+0x28/0x50 [drm_kms_helper]
20:22:40:[   68.789430]  [<ffffffff8137cdb2>] ? fbcon_init+0x4c2/0x540
20:22:40:[   68.789444]  [<ffffffff8140520f>] ? visual_init+0xcf/0x130
20:22:40:[   68.789445]  [<ffffffff81407689>] ? do_bind_con_driver+0x149/0x320
20:22:40:[   68.789447]  [<ffffffff81407b87>] ? do_take_over_console+0x137/0x1a0
20:22:40:[   68.789448]  [<ffffffff813784a3>] ? do_fbcon_takeover+0x53/0xb0
20:22:40:[   68.789458]  [<ffffffff8109ab04>] ? notifier_call_chain+0x44/0x70
20:22:40:[   68.789460]  [<ffffffff8109ae07>] ? __blocking_notifier_call_chain+0x47/0x60
20:22:40:[   68.789464]  [<ffffffff81384a29>] ? register_framebuffer+0x1d9/0x300
20:22:40:[   68.789468]  [<ffffffffa0240eea>] ? drm_fb_helper_initial_config+0x25a/0x3c0 [drm_kms_helper]
20:22:40:[   68.789470]  [<ffffffffa0269580>] ? cirrus_modeset_init+0x180/0x1d0 [cirrus]
20:22:40:[   68.789476]  [<ffffffffa0268520>] ? cirrus_driver_load+0xb0/0xf0 [cirrus]
20:22:40:[   68.789483]  [<ffffffffa015ad6c>] ? drm_dev_register+0x9c/0xb0 [drm]
20:22:40:[   68.789490]  [<ffffffffa015d158>] ? drm_get_pci_dev+0x88/0x1e0 [drm]
20:22:40:[   68.789495]  [<ffffffff81357bff>] ? local_pci_probe+0x3f/0xa0
20:22:40:[   68.789497]  [<ffffffff81359024>] ? pci_device_probe+0xd4/0x120
20:22:40:[   68.789507]  [<ffffffff8144f467>] ? driver_probe_device+0x1f7/0x420
20:22:40:[   68.789509]  [<ffffffff8144f70b>] ? __driver_attach+0x7b/0x80
20:22:40:[   68.789510]  [<ffffffff8144f690>] ? driver_probe_device+0x420/0x420
20:22:40:[   68.789512]  [<ffffffff8144d378>] ? bus_for_each_dev+0x58/0x90
20:22:40:[   68.789514]  [<ffffffff8144e8c9>] ? bus_add_driver+0x1c9/0x280
20:22:40:[   68.789516]  [<ffffffffa01fc000>] ? 0xffffffffa01fc000
20:22:40:[   68.789517]  [<ffffffff8145007b>] ? driver_register+0x5b/0xd0
20:22:40:[   68.789526]  [<ffffffff81002138>] ? do_one_initcall+0xc8/0x1f0
20:22:40:[   68.789539]  [<ffffffff811835a8>] ? do_init_module+0x5a/0x1d7
20:22:40:[   68.789549]  [<ffffffff81103fa6>] ? load_module+0x1366/0x1c50
20:22:40:[   68.789550]  [<ffffffff811007a0>] ? __symbol_put+0x30/0x30
20:22:40:[   68.789552]  [<ffffffff81104a40>] ? SYSC_finit_module+0x70/0xa0
20:22:40:[   68.789564]  [<ffffffff815e402e>] ? entry_SYSCALL_64_fastpath+0x12/0x6d
20:22:40:[   68.789580] Code: b8 01 01 01 01 01 01 01 01 48 0f af c1 41 89 f9 41 83 e1 07 75 70 48 89 d1 48 c1 e9 06 74 39 66 0f 1f 84 00 00 00 00 00 48 ff c9 <48> 89 07 48 89 47 08 48 89 47 10 48 89 47 18 48 89 47 20 48 89 
20:22:40:[   68.789591] Kernel panic - not syncing: softlockup: hung tasks
20:22:40:[   68.789597] CPU: 0 PID: 187 Comm: systemd-udevd Tainted: G        W    L     4.4.59-92.17-default #1
20:22:40:[   68.789597] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
20:22:40:[   68.789599]  0000000000000000 ffffffff8130f110 ffffffff81860d69 ffff880033e03ed0
20:22:40:[   68.789599]  ffffffff81182771 0000000000000008 ffff880033e03ee0 ffff880033e03e80
20:22:40:[   68.789600]  ffffffff82008980 0000000000000000 0000000000000001 0000000000000000
20:22:40:[   68.789601] Call Trace:
20:22:40:[   68.789621]  [<ffffffff81019a99>] dump_trace+0x59/0x310
20:22:40:[   68.789624]  [<ffffffff81019e3a>] show_stack_log_lvl+0xea/0x170
20:22:40:[   68.789626]  [<ffffffff8101abc1>] show_stack+0x21/0x40
20:22:40:[   68.789633]  [<ffffffff8130f110>] dump_stack+0x5c/0x7c
20:22:40:[   68.789642]  [<ffffffff81182771>] panic+0xd2/0x219
20:22:40:[   68.789657]  [<ffffffff81136359>] watchdog_timer_fn+0x1d9/0x1e0
20:22:40:[   68.789672]  [<ffffffff810eadaa>] __hrtimer_run_queues+0xea/0x260
20:22:40:[   68.789678]  [<ffffffff810eb1e9>] hrtimer_interrupt+0x99/0x190
20:22:40:[   68.789682]  [<ffffffff815e6ce9>] smp_apic_timer_interrupt+0x39/0x50
20:22:40:[   68.789691]  [<ffffffff815e4dbc>] apic_timer_interrupt+0x8c/0xa0
20:22:40:[   68.790950] DWARF2 unwinder stuck at apic_timer_interrupt+0x8c/0xa0
20:22:40:[   68.790950] 
20:22:40:[   68.790950] Leftover inexact backtrace:
20:22:40:[   68.790950] 
20:22:40:[   68.790955]  <IRQ>  <EOI>  [<ffffffff8131cb53>] ? memset_orig+0x33/0xb0
20:22:40:[   68.790959]  [<ffffffffa01e148c>] ? ttm_bo_move_memcpy+0x5dc/0x690 [ttm]
20:22:40:[   68.790962]  [<ffffffffa01dec58>] ? ttm_bo_handle_move_mem+0x258/0x590 [ttm]
20:22:40:[   68.790964]  [<ffffffffa01df5d4>] ? ttm_bo_mem_space+0x134/0x390 [ttm]
20:22:40:[   68.790967]  [<ffffffffa01dfbde>] ? ttm_bo_validate+0x11e/0x130 [ttm]
20:22:40:[   68.790969]  [<ffffffffa026a5e6>] ? cirrus_bo_pin+0x76/0xa0 [cirrus]
20:22:40:[   68.790971]  [<ffffffffa0268c37>] ? cirrus_crtc_do_set_base.isra.9.constprop.11+0xe7/0x3a0 [cirrus]
20:22:40:[   68.790973]  [<ffffffffa026932e>] ? cirrus_crtc_mode_set+0x43e/0x4a0 [cirrus]
20:22:40:[   68.790976]  [<ffffffffa0232623>] ? drm_crtc_helper_set_mode+0x293/0x4b0 [drm_kms_helper]
20:22:40:[   68.790980]  [<ffffffffa023374d>] ? drm_crtc_helper_set_config+0x91d/0xb50 [drm_kms_helper]
20:22:40:[   68.790989]  [<ffffffffa01612c0>] ? drm_mode_set_config_internal+0x60/0xf0 [drm]
20:22:40:[   68.790993]  [<ffffffffa023ea4d>] ? restore_fbdev_mode+0xad/0x250 [drm_kms_helper]
20:22:40:[   68.791002]  [<ffffffffa016fe20>] ? drm_modeset_lock_all_ctx+0x90/0xa0 [drm]
20:22:40:[   68.791006]  [<ffffffffa0240bfe>] ? drm_fb_helper_restore_fbdev_mode_unlocked+0x2e/0x70 [drm_kms_helper]
20:22:40:[   68.791009]  [<ffffffffa0240c68>] ? drm_fb_helper_set_par+0x28/0x50 [drm_kms_helper]
20:22:40:[   68.791010]  [<ffffffff8137cdb2>] ? fbcon_init+0x4c2/0x540
20:22:40:[   68.791011]  [<ffffffff8140520f>] ? visual_init+0xcf/0x130
20:22:40:[   68.791012]  [<ffffffff81407689>] ? do_bind_con_driver+0x149/0x320
20:22:40:[   68.791014]  [<ffffffff81407b87>] ? do_take_over_console+0x137/0x1a0
20:22:40:[   68.791015]  [<ffffffff813784a3>] ? do_fbcon_takeover+0x53/0xb0
20:22:40:[   68.791016]  [<ffffffff8109ab04>] ? notifier_call_chain+0x44/0x70
20:22:40:[   68.791017]  [<ffffffff8109ae07>] ? __blocking_notifier_call_chain+0x47/0x60
20:22:40:[   68.791018]  [<ffffffff81384a29>] ? register_framebuffer+0x1d9/0x300
20:22:40:[   68.791021]  [<ffffffffa0240eea>] ? drm_fb_helper_initial_config+0x25a/0x3c0 [drm_kms_helper]
20:22:40:[   68.791023]  [<ffffffffa0269580>] ? cirrus_modeset_init+0x180/0x1d0 [cirrus]
20:22:40:[   68.791025]  [<ffffffffa0268520>] ? cirrus_driver_load+0xb0/0xf0 [cirrus]
20:22:40:[   68.791032]  [<ffffffffa015ad6c>] ? drm_dev_register+0x9c/0xb0 [drm]
20:22:40:[   68.791039]  [<ffffffffa015d158>] ? drm_get_pci_dev+0x88/0x1e0 [drm]
20:22:40:[   68.791040]  [<ffffffff81357bff>] ? local_pci_probe+0x3f/0xa0
20:22:40:[   68.791041]  [<ffffffff81359024>] ? pci_device_probe+0xd4/0x120
20:22:40:[   68.791043]  [<ffffffff8144f467>] ? driver_probe_device+0x1f7/0x420
20:22:40:[   68.791045]  [<ffffffff8144f70b>] ? __driver_attach+0x7b/0x80
20:22:40:[   68.791046]  [<ffffffff8144f690>] ? driver_probe_device+0x420/0x420
20:22:40:[   68.791047]  [<ffffffff8144d378>] ? bus_for_each_dev+0x58/0x90
20:22:40:[   68.791050]  [<ffffffff8144e8c9>] ? bus_add_driver+0x1c9/0x280
20:22:40:[   68.791051]  [<ffffffffa01fc000>] ? 0xffffffffa01fc000
20:22:40:[   68.791052]  [<ffffffff8145007b>] ? driver_register+0x5b/0xd0
20:22:40:[   68.791054]  [<ffffffff81002138>] ? do_one_initcall+0xc8/0x1f0
20:22:40:[   68.791055]  [<ffffffff811835a8>] ? do_init_module+0x5a/0x1d7
20:30:05:[   68.791057]  [<ffffffff81103fa6>] ? load_module+0x1366/0x1c50
20:30:05:[   68.791058]  [<ffffffff811007a0>] ? __symbol_put+0x30/0x30
20:30:05:[   68.791060]  [<ffffffff81104a40>] ? SYSC_finit_module+0x70/0xa0
20:30:05:[   68.791062]  [<ffffffff815e402e>] ? entry_SYSCALL_64_fastpath+0x12/0x6d
20:30:06:[   68.791114] Kernel Offset: disabled

Whamcloud Community JIRA

[LU-9705] sanity-benchmark test_iozone: client hit Kernel panic - not syncing: softlockup: hung tasks