Apr 24 00:33:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client f50d7e88-e046-b55a-0fe3-13cbe95d417b (at 10.8.8.33@o2ib6) reconnecting Apr 24 00:33:31 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages Apr 24 00:42:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to ef4ceb90-5603-14d7-21e5-f4e7d223a0a1 (at 10.8.8.33@o2ib6) Apr 24 00:42:16 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages Apr 24 00:42:58 fir-md1-s1 kernel: list passed to list_sort() too long for efficiency Apr 24 00:43:07 fir-md1-s1 kernel: Lustre: 21207:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8b37931c4800 x1631534610856048/t0(0) o101->40edf202-1dd9-d482-e1d9-d8df453c34c1@10.9.102.4@o2ib4:12/0 lens 568/0 e 1 to 0 dl 1556091792 ref 2 fl Interpret:/0/ffffffff rc 0/-1 Apr 24 00:43:07 fir-md1-s1 kernel: Lustre: 21207:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages Apr 24 00:43:19 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#40 stuck for 23s! [mdt00_090:22029] Apr 24 00:43:19 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif Apr 24 00:43:19 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas Apr 24 00:43:19 fir-md1-s1 kernel: CPU: 40 PID: 22029 Comm: mdt00_090 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 Apr 24 00:43:19 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 Apr 24 00:43:19 fir-md1-s1 kernel: task: ffff8b7324050000 ti: ffff8b7324ab0000 task.ti: ffff8b7324ab0000 Apr 24 00:43:19 fir-md1-s1 kernel: RIP: 0010:[] [] ldiskfs_inode_touch_time_cmp+0xd/0x90 [ldiskfs] Apr 24 00:43:19 fir-md1-s1 kernel: RSP: 0018:ffff8b7324ab3310 EFLAGS: 00000282 Apr 24 00:43:19 fir-md1-s1 kernel: RAX: 8000040400080000 RBX: ffffffff9c3e3002 RCX: 00000001043969bf Apr 24 00:43:19 fir-md1-s1 kernel: RDX: ffff8b4b1db2d278 RSI: ffff8b3d335b3138 RDI: 0000000000000000 Apr 24 00:43:19 fir-md1-s1 kernel: RBP: ffff8b7324ab3360 R08: 000000000000000a R09: 0000000000000000 Apr 24 00:43:19 fir-md1-s1 kernel: R10: 0000000000000fbe R11: ffff8b7324ab306e R12: ffff8b7324ab32f0 Apr 24 00:43:19 fir-md1-s1 kernel: R13: 0000000000000006 R14: 0000000000000032 R15: 0000000000000000 Apr 24 00:43:19 fir-md1-s1 kernel: FS: 00007f2994e3b780(0000) GS:ffff8b433f080000(0000) knlGS:0000000000000000 Apr 24 00:43:19 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 24 00:43:19 fir-md1-s1 kernel: CR2: 00007f2994e4c000 CR3: 0000000dd2c10000 CR4: 00000000003407e0 Apr 24 00:43:19 fir-md1-s1 kernel: Call Trace: Apr 24 00:43:19 fir-md1-s1 kernel: [] ? merge+0x62/0xc0 Apr 24 00:43:19 fir-md1-s1 kernel: [] ? ldiskfs_init_inode_table+0x410/0x410 [ldiskfs] Apr 24 00:43:19 fir-md1-s1 kernel: [] list_sort+0x9b/0x250 Apr 24 00:43:19 fir-md1-s1 kernel: [] __ldiskfs_es_shrink+0x1ce/0x2a0 [ldiskfs] Apr 24 00:43:19 fir-md1-s1 kernel: [] ldiskfs_es_shrink+0xb4/0x130 [ldiskfs] Apr 24 00:43:19 fir-md1-s1 kernel: [] shrink_slab+0x175/0x340 Apr 24 00:43:19 fir-md1-s1 kernel: [] ? zone_watermark_ok+0x1f/0x30 Apr 24 00:43:19 fir-md1-s1 kernel: [] ? compaction_suitable+0xa3/0xb0 Apr 24 00:43:19 fir-md1-s1 kernel: [] zone_reclaim+0x1d1/0x2f0 Apr 24 00:43:19 fir-md1-s1 kernel: [] get_page_from_freelist+0x87b/0xa70 Apr 24 00:43:19 fir-md1-s1 kernel: [] ? list_del+0xd/0x30 Apr 24 00:43:19 fir-md1-s1 kernel: [] ? pointer.isra.19+0x1c9/0x4d0 Apr 24 00:43:19 fir-md1-s1 kernel: [] ? vsnprintf+0x234/0x6a0 Apr 24 00:43:19 fir-md1-s1 kernel: [] __alloc_pages_nodemask+0x176/0x420 Apr 24 00:43:19 fir-md1-s1 kernel: [] alloc_pages_current+0x98/0x110 Apr 24 00:43:19 fir-md1-s1 kernel: [] __get_free_pages+0xe/0x40 Apr 24 00:43:19 fir-md1-s1 kernel: [] kmalloc_order_trace+0x2e/0xa0 Apr 24 00:43:19 fir-md1-s1 kernel: [] __kmalloc+0x211/0x230 Apr 24 00:43:19 fir-md1-s1 kernel: [] null_alloc_rs+0x16d/0x340 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] sptlrpc_svc_alloc_rs+0x66/0x350 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] ? mdt_root_squash+0x21/0x430 [mdt] Apr 24 00:43:19 fir-md1-s1 kernel: [] lustre_pack_reply_v2+0x8a/0x280 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] lustre_pack_reply+0x11/0x20 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] req_capsule_server_pack+0x43/0xf0 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] mdt_getxattr+0x71c/0x12e0 [mdt] Apr 24 00:43:19 fir-md1-s1 kernel: [] ? mdt_object_lock_internal+0x70/0x3e0 [mdt] Apr 24 00:43:19 fir-md1-s1 kernel: [] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] mdt_intent_getxattr+0xc5/0x270 [mdt] Apr 24 00:43:19 fir-md1-s1 kernel: [] mdt_intent_policy+0x2e8/0xd00 [mdt] Apr 24 00:43:19 fir-md1-s1 kernel: [] ? mdt_intent_getattr+0x480/0x480 [mdt] Apr 24 00:43:19 fir-md1-s1 kernel: [] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs] Apr 24 00:43:19 fir-md1-s1 kernel: [] ? cfs_hash_add+0xbe/0x1a0 [libcfs] Apr 24 00:43:19 fir-md1-s1 kernel: [] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:19 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] ? default_wake_function+0x12/0x20 Apr 24 00:43:19 fir-md1-s1 kernel: [] ? __wake_up_common+0x5b/0x90 Apr 24 00:43:19 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Apr 24 00:43:19 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Apr 24 00:43:19 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:19 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Apr 24 00:43:19 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:19 fir-md1-s1 kernel: Code: ff 8d 4a 01 89 d0 f0 0f b1 0f 39 d0 0f 84 fb fd ff ff 89 c2 eb e2 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 86 e8 fc ff ff <48> 89 e5 48 c1 e8 2b a8 01 74 15 48 8b 8a e8 fc ff ff b8 01 00 Apr 24 00:43:22 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [mdt00_039:21591] Apr 24 00:43:22 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [mdt_io01_084:22135] Apr 24 00:43:22 fir-md1-s1 kernel: Modules linked in: Apr 24 00:43:22 fir-md1-s1 kernel: Modules linked in: Apr 24 00:43:22 fir-md1-s1 kernel: osp(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mdd(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lod(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mdt(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lfsck(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mgs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mgc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: osd_ldiskfs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lquota(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ldiskfs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lustre(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lmv(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mdc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: osc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lov(OE) Apr 24 00:43:22 fir-md1-s1 kernel: fid(OE) Apr 24 00:43:22 fir-md1-s1 kernel: fld(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ko2iblnd(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ptlrpc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: obdclass(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lnet(OE) Apr 24 00:43:22 fir-md1-s1 kernel: libcfs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: rpcsec_gss_krb5 Apr 24 00:43:22 fir-md1-s1 kernel: auth_rpcgss Apr 24 00:43:22 fir-md1-s1 kernel: nfsv4 Apr 24 00:43:22 fir-md1-s1 kernel: dns_resolver Apr 24 00:43:22 fir-md1-s1 kernel: nfs Apr 24 00:43:22 fir-md1-s1 kernel: lockd Apr 24 00:43:22 fir-md1-s1 kernel: grace Apr 24 00:43:22 fir-md1-s1 kernel: fscache Apr 24 00:43:22 fir-md1-s1 kernel: rdma_ucm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_ucm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: rdma_cm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: iw_cm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_ipoib(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_cm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_umad(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_fpga_tools(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_en(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_ib(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_core(OE) Apr 24 00:43:22 fir-md1-s1 kernel: dell_rbu Apr 24 00:43:22 fir-md1-s1 kernel: sunrpc Apr 24 00:43:22 fir-md1-s1 kernel: vfat Apr 24 00:43:22 fir-md1-s1 kernel: fat Apr 24 00:43:22 fir-md1-s1 kernel: dm_round_robin Apr 24 00:43:22 fir-md1-s1 kernel: amd64_edac_mod Apr 24 00:43:22 fir-md1-s1 kernel: edac_mce_amd Apr 24 00:43:22 fir-md1-s1 kernel: kvm_amd Apr 24 00:43:22 fir-md1-s1 kernel: kvm Apr 24 00:43:22 fir-md1-s1 kernel: irqbypass Apr 24 00:43:22 fir-md1-s1 kernel: crc32_pclmul Apr 24 00:43:22 fir-md1-s1 kernel: ghash_clmulni_intel Apr 24 00:43:22 fir-md1-s1 kernel: aesni_intel Apr 24 00:43:22 fir-md1-s1 kernel: lrw Apr 24 00:43:22 fir-md1-s1 kernel: gf128mul Apr 24 00:43:22 fir-md1-s1 kernel: glue_helper Apr 24 00:43:22 fir-md1-s1 kernel: ablk_helper Apr 24 00:43:22 fir-md1-s1 kernel: cryptd Apr 24 00:43:22 fir-md1-s1 kernel: dcdbas Apr 24 00:43:22 fir-md1-s1 kernel: ses Apr 24 00:43:22 fir-md1-s1 kernel: enclosure Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_si Apr 24 00:43:22 fir-md1-s1 kernel: pcspkr Apr 24 00:43:22 fir-md1-s1 kernel: dm_multipath Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_devintf Apr 24 00:43:22 fir-md1-s1 kernel: dm_mod Apr 24 00:43:22 fir-md1-s1 kernel: ccp Apr 24 00:43:22 fir-md1-s1 kernel: sg Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_msghandler Apr 24 00:43:22 fir-md1-s1 kernel: k10temp Apr 24 00:43:22 fir-md1-s1 kernel: i2c_piix4 Apr 24 00:43:22 fir-md1-s1 kernel: acpi_power_meter Apr 24 00:43:22 fir-md1-s1 kernel: knem(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ip_tables Apr 24 00:43:22 fir-md1-s1 kernel: ext4 Apr 24 00:43:22 fir-md1-s1 kernel: mbcache Apr 24 00:43:22 fir-md1-s1 kernel: jbd2 Apr 24 00:43:22 fir-md1-s1 kernel: sd_mod Apr 24 00:43:22 fir-md1-s1 kernel: crc_t10dif Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_generic Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_ib(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_uverbs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_core(OE) Apr 24 00:43:22 fir-md1-s1 kernel: i2c_algo_bit Apr 24 00:43:22 fir-md1-s1 kernel: drm_kms_helper Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_core(OE) Apr 24 00:43:22 fir-md1-s1 kernel: syscopyarea Apr 24 00:43:22 fir-md1-s1 kernel: mlxfw(OE) Apr 24 00:43:22 fir-md1-s1 kernel: sysfillrect Apr 24 00:43:22 fir-md1-s1 kernel: devlink Apr 24 00:43:22 fir-md1-s1 kernel: sysimgblt Apr 24 00:43:22 fir-md1-s1 kernel: fb_sys_fops Apr 24 00:43:22 fir-md1-s1 kernel: mlx_compat(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ahci Apr 24 00:43:22 fir-md1-s1 kernel: ttm Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_pclmul Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_common Apr 24 00:43:22 fir-md1-s1 kernel: libahci Apr 24 00:43:22 fir-md1-s1 kernel: drm Apr 24 00:43:22 fir-md1-s1 kernel: tg3 Apr 24 00:43:22 fir-md1-s1 kernel: crc32c_intel Apr 24 00:43:22 fir-md1-s1 kernel: ptp Apr 24 00:43:22 fir-md1-s1 kernel: libata Apr 24 00:43:22 fir-md1-s1 kernel: megaraid_sas Apr 24 00:43:22 fir-md1-s1 kernel: drm_panel_orientation_quirks Apr 24 00:43:22 fir-md1-s1 kernel: pps_core Apr 24 00:43:22 fir-md1-s1 kernel: mpt3sas(OE) Apr 24 00:43:22 fir-md1-s1 kernel: raid_class Apr 24 00:43:22 fir-md1-s1 kernel: scsi_transport_sas Apr 24 00:43:22 fir-md1-s1 kernel: Apr 24 00:43:22 fir-md1-s1 kernel: CPU: 1 PID: 22135 Comm: mdt_io01_084 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 Apr 24 00:43:22 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 Apr 24 00:43:22 fir-md1-s1 kernel: task: ffff8b732d24a080 ti: ffff8b732d254000 task.ti: ffff8b732d254000 Apr 24 00:43:22 fir-md1-s1 kernel: RIP: 0010:[] Apr 24 00:43:22 fir-md1-s1 kernel: [] native_queued_spin_lock_slowpath+0x122/0x200 Apr 24 00:43:22 fir-md1-s1 kernel: RSP: 0018:ffff8b732d257800 EFLAGS: 00000246 Apr 24 00:43:22 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b3793a33da8 RCX: 0000000000090000 Apr 24 00:43:22 fir-md1-s1 kernel: RDX: ffff8b633f69b780 RSI: 0000000000510101 RDI: ffff8b7339b45480 Apr 24 00:43:22 fir-md1-s1 kernel: RBP: ffff8b732d257800 R08: ffff8b533f61b780 R09: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: R10: ffff8b533f61f140 R11: fffff657da776800 R12: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: R13: ffff8b732d2577a0 R14: ffff8b3793a33b18 R15: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: FS: 00007f43a2cd8700(0000) GS:ffff8b533f600000(0000) knlGS:0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 24 00:43:22 fir-md1-s1 kernel: CR2: 00007f43a2dcf000 CR3: 0000000dd2c10000 CR4: 00000000003407e0 Apr 24 00:43:22 fir-md1-s1 kernel: Call Trace: Apr 24 00:43:22 fir-md1-s1 kernel: [] queued_spin_lock_slowpath+0xb/0xf Apr 24 00:43:22 fir-md1-s1 kernel: [] _raw_spin_lock+0x20/0x30 Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ktime_get_ts64+0x52/0xf0 Apr 24 00:43:22 fir-md1-s1 kernel: [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] mdt_obd_preprw+0x637/0x1060 [mdt] Apr 24 00:43:22 fir-md1-s1 kernel: [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? update_curr+0x14c/0x1e0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? account_entity_dequeue+0xae/0xd0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? default_wake_function+0x12/0x20 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? __wake_up_common+0x5b/0x90 Apr 24 00:43:22 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:22 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:22 fir-md1-s1 kernel: Code: Apr 24 00:43:22 fir-md1-s1 kernel: 13 Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: c1 Apr 24 00:43:22 fir-md1-s1 kernel: ea Apr 24 00:43:22 fir-md1-s1 kernel: 0d Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: 98 Apr 24 00:43:22 fir-md1-s1 kernel: 83 Apr 24 00:43:22 fir-md1-s1 kernel: e2 Apr 24 00:43:22 fir-md1-s1 kernel: 30 Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: 81 Apr 24 00:43:22 fir-md1-s1 kernel: c2 Apr 24 00:43:22 fir-md1-s1 kernel: 80 Apr 24 00:43:22 fir-md1-s1 kernel: b7 Apr 24 00:43:22 fir-md1-s1 kernel: 01 Apr 24 00:43:22 fir-md1-s1 kernel: 00 Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: 03 Apr 24 00:43:22 fir-md1-s1 kernel: 14 Apr 24 00:43:22 fir-md1-s1 kernel: c5 Apr 24 00:43:22 fir-md1-s1 kernel: 60 Apr 24 00:43:22 fir-md1-s1 kernel: b9 Apr 24 00:43:22 fir-md1-s1 kernel: 14 Apr 24 00:43:22 fir-md1-s1 kernel: 9c Apr 24 00:43:22 fir-md1-s1 kernel: 4c Apr 24 00:43:22 fir-md1-s1 kernel: 89 Apr 24 00:43:22 fir-md1-s1 kernel: 02 Apr 24 00:43:22 fir-md1-s1 kernel: 41 Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: 40 Apr 24 00:43:22 fir-md1-s1 kernel: 08 Apr 24 00:43:22 fir-md1-s1 kernel: 85 Apr 24 00:43:22 fir-md1-s1 kernel: c0 Apr 24 00:43:22 fir-md1-s1 kernel: 75 Apr 24 00:43:22 fir-md1-s1 kernel: 0f Apr 24 00:43:22 fir-md1-s1 kernel: 0f Apr 24 00:43:22 fir-md1-s1 kernel: 1f Apr 24 00:43:22 fir-md1-s1 kernel: 44 Apr 24 00:43:22 fir-md1-s1 kernel: 00 Apr 24 00:43:22 fir-md1-s1 kernel: 00 Apr 24 00:43:22 fir-md1-s1 kernel: f3 Apr 24 00:43:22 fir-md1-s1 kernel: 90 Apr 24 00:43:22 fir-md1-s1 kernel: <41> Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: 40 Apr 24 00:43:22 fir-md1-s1 kernel: 08 Apr 24 00:43:22 fir-md1-s1 kernel: 85 Apr 24 00:43:22 fir-md1-s1 kernel: c0 Apr 24 00:43:22 fir-md1-s1 kernel: 74 Apr 24 00:43:22 fir-md1-s1 kernel: f6 Apr 24 00:43:22 fir-md1-s1 kernel: 4d Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: 08 Apr 24 00:43:22 fir-md1-s1 kernel: 4d Apr 24 00:43:22 fir-md1-s1 kernel: 85 Apr 24 00:43:22 fir-md1-s1 kernel: c9 Apr 24 00:43:22 fir-md1-s1 kernel: 74 Apr 24 00:43:22 fir-md1-s1 kernel: 04 Apr 24 00:43:22 fir-md1-s1 kernel: 41 Apr 24 00:43:22 fir-md1-s1 kernel: 0f Apr 24 00:43:22 fir-md1-s1 kernel: 18 Apr 24 00:43:22 fir-md1-s1 kernel: 09 Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: Apr 24 00:43:22 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [mdt_io00_022:21608] Apr 24 00:43:22 fir-md1-s1 kernel: Modules linked in: Apr 24 00:43:22 fir-md1-s1 kernel: osp(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mdd(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lod(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mdt(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lfsck(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mgs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mgc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: osd_ldiskfs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lquota(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ldiskfs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lustre(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lmv(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mdc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: osc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lov(OE) Apr 24 00:43:22 fir-md1-s1 kernel: fid(OE) Apr 24 00:43:22 fir-md1-s1 kernel: fld(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ko2iblnd(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ptlrpc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: obdclass(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lnet(OE) Apr 24 00:43:22 fir-md1-s1 kernel: libcfs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: rpcsec_gss_krb5 Apr 24 00:43:22 fir-md1-s1 kernel: auth_rpcgss Apr 24 00:43:22 fir-md1-s1 kernel: nfsv4 Apr 24 00:43:22 fir-md1-s1 kernel: dns_resolver Apr 24 00:43:22 fir-md1-s1 kernel: nfs Apr 24 00:43:22 fir-md1-s1 kernel: lockd Apr 24 00:43:22 fir-md1-s1 kernel: grace Apr 24 00:43:22 fir-md1-s1 kernel: fscache Apr 24 00:43:22 fir-md1-s1 kernel: rdma_ucm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_ucm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: rdma_cm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: iw_cm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_ipoib(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_cm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_umad(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_fpga_tools(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_en(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_ib(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_core(OE) Apr 24 00:43:22 fir-md1-s1 kernel: dell_rbu Apr 24 00:43:22 fir-md1-s1 kernel: sunrpc Apr 24 00:43:22 fir-md1-s1 kernel: vfat Apr 24 00:43:22 fir-md1-s1 kernel: fat Apr 24 00:43:22 fir-md1-s1 kernel: dm_round_robin Apr 24 00:43:22 fir-md1-s1 kernel: amd64_edac_mod Apr 24 00:43:22 fir-md1-s1 kernel: edac_mce_amd Apr 24 00:43:22 fir-md1-s1 kernel: kvm_amd Apr 24 00:43:22 fir-md1-s1 kernel: kvm Apr 24 00:43:22 fir-md1-s1 kernel: irqbypass Apr 24 00:43:22 fir-md1-s1 kernel: crc32_pclmul Apr 24 00:43:22 fir-md1-s1 kernel: ghash_clmulni_intel Apr 24 00:43:22 fir-md1-s1 kernel: aesni_intel Apr 24 00:43:22 fir-md1-s1 kernel: lrw Apr 24 00:43:22 fir-md1-s1 kernel: gf128mul Apr 24 00:43:22 fir-md1-s1 kernel: glue_helper Apr 24 00:43:22 fir-md1-s1 kernel: ablk_helper Apr 24 00:43:22 fir-md1-s1 kernel: cryptd Apr 24 00:43:22 fir-md1-s1 kernel: dcdbas Apr 24 00:43:22 fir-md1-s1 kernel: ses Apr 24 00:43:22 fir-md1-s1 kernel: enclosure Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_si Apr 24 00:43:22 fir-md1-s1 kernel: pcspkr Apr 24 00:43:22 fir-md1-s1 kernel: dm_multipath Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_devintf Apr 24 00:43:22 fir-md1-s1 kernel: dm_mod Apr 24 00:43:22 fir-md1-s1 kernel: ccp Apr 24 00:43:22 fir-md1-s1 kernel: sg Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_msghandler Apr 24 00:43:22 fir-md1-s1 kernel: k10temp Apr 24 00:43:22 fir-md1-s1 kernel: i2c_piix4 Apr 24 00:43:22 fir-md1-s1 kernel: acpi_power_meter Apr 24 00:43:22 fir-md1-s1 kernel: knem(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ip_tables Apr 24 00:43:22 fir-md1-s1 kernel: ext4 Apr 24 00:43:22 fir-md1-s1 kernel: mbcache Apr 24 00:43:22 fir-md1-s1 kernel: jbd2 Apr 24 00:43:22 fir-md1-s1 kernel: sd_mod Apr 24 00:43:22 fir-md1-s1 kernel: crc_t10dif Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_generic Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_ib(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_uverbs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_core(OE) Apr 24 00:43:22 fir-md1-s1 kernel: i2c_algo_bit Apr 24 00:43:22 fir-md1-s1 kernel: drm_kms_helper Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_core(OE) Apr 24 00:43:22 fir-md1-s1 kernel: syscopyarea Apr 24 00:43:22 fir-md1-s1 kernel: mlxfw(OE) Apr 24 00:43:22 fir-md1-s1 kernel: sysfillrect Apr 24 00:43:22 fir-md1-s1 kernel: devlink Apr 24 00:43:22 fir-md1-s1 kernel: sysimgblt Apr 24 00:43:22 fir-md1-s1 kernel: fb_sys_fops Apr 24 00:43:22 fir-md1-s1 kernel: mlx_compat(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ahci Apr 24 00:43:22 fir-md1-s1 kernel: ttm Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_pclmul Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_common Apr 24 00:43:22 fir-md1-s1 kernel: libahci Apr 24 00:43:22 fir-md1-s1 kernel: drm Apr 24 00:43:22 fir-md1-s1 kernel: tg3 Apr 24 00:43:22 fir-md1-s1 kernel: crc32c_intel Apr 24 00:43:22 fir-md1-s1 kernel: ptp Apr 24 00:43:22 fir-md1-s1 kernel: libata Apr 24 00:43:22 fir-md1-s1 kernel: megaraid_sas Apr 24 00:43:22 fir-md1-s1 kernel: drm_panel_orientation_quirks Apr 24 00:43:22 fir-md1-s1 kernel: pps_core Apr 24 00:43:22 fir-md1-s1 kernel: mpt3sas(OE) Apr 24 00:43:22 fir-md1-s1 kernel: raid_class Apr 24 00:43:22 fir-md1-s1 kernel: scsi_transport_sas Apr 24 00:43:22 fir-md1-s1 kernel: Apr 24 00:43:22 fir-md1-s1 kernel: CPU: 4 PID: 21608 Comm: mdt_io00_022 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 Apr 24 00:43:22 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 Apr 24 00:43:22 fir-md1-s1 kernel: task: ffff8b7321131040 ti: ffff8b7327a18000 task.ti: ffff8b7327a18000 Apr 24 00:43:22 fir-md1-s1 kernel: RIP: 0010:[] Apr 24 00:43:22 fir-md1-s1 kernel: [] native_queued_spin_lock_slowpath+0x122/0x200 Apr 24 00:43:22 fir-md1-s1 kernel: RSP: 0018:ffff8b7327a1b800 EFLAGS: 00000246 Apr 24 00:43:22 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fb62b558 RCX: 0000000000210000 Apr 24 00:43:22 fir-md1-s1 kernel: RDX: ffff8b533f69b780 RSI: 0000000000490101 RDI: ffff8b7339b45480 Apr 24 00:43:22 fir-md1-s1 kernel: RBP: ffff8b7327a1b800 R08: ffff8b433ee5b780 R09: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: R10: ffff8b433ee5f140 R11: fffff657dc2bc200 R12: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: R13: ffff8b7327a1b7a0 R14: ffff8b48fb62b2c8 R15: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: FS: 00007fa2ad0f3700(0000) GS:ffff8b433ee40000(0000) knlGS:0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 24 00:43:22 fir-md1-s1 kernel: CR2: 00007f232944eb04 CR3: 0000000dd2c10000 CR4: 00000000003407e0 Apr 24 00:43:22 fir-md1-s1 kernel: Call Trace: Apr 24 00:43:22 fir-md1-s1 kernel: [] queued_spin_lock_slowpath+0xb/0xf Apr 24 00:43:22 fir-md1-s1 kernel: [] _raw_spin_lock+0x20/0x30 Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ktime_get_ts64+0x52/0xf0 Apr 24 00:43:22 fir-md1-s1 kernel: [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] mdt_obd_preprw+0x637/0x1060 [mdt] Apr 24 00:43:22 fir-md1-s1 kernel: [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? update_curr+0x14c/0x1e0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? account_entity_dequeue+0xae/0xd0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? default_wake_function+0x12/0x20 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? __wake_up_common+0x5b/0x90 Apr 24 00:43:22 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:22 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:22 fir-md1-s1 kernel: Code: Apr 24 00:43:22 fir-md1-s1 kernel: 13 Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: c1 Apr 24 00:43:22 fir-md1-s1 kernel: ea Apr 24 00:43:22 fir-md1-s1 kernel: 0d Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: 98 Apr 24 00:43:22 fir-md1-s1 kernel: 83 Apr 24 00:43:22 fir-md1-s1 kernel: e2 Apr 24 00:43:22 fir-md1-s1 kernel: 30 Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: 81 Apr 24 00:43:22 fir-md1-s1 kernel: c2 Apr 24 00:43:22 fir-md1-s1 kernel: 80 Apr 24 00:43:22 fir-md1-s1 kernel: b7 Apr 24 00:43:22 fir-md1-s1 kernel: 01 Apr 24 00:43:22 fir-md1-s1 kernel: 00 Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: 03 Apr 24 00:43:22 fir-md1-s1 kernel: 14 Apr 24 00:43:22 fir-md1-s1 kernel: c5 Apr 24 00:43:22 fir-md1-s1 kernel: 60 Apr 24 00:43:22 fir-md1-s1 kernel: b9 Apr 24 00:43:22 fir-md1-s1 kernel: 14 Apr 24 00:43:22 fir-md1-s1 kernel: 9c Apr 24 00:43:22 fir-md1-s1 kernel: 4c Apr 24 00:43:22 fir-md1-s1 kernel: 89 Apr 24 00:43:22 fir-md1-s1 kernel: 02 Apr 24 00:43:22 fir-md1-s1 kernel: 41 Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: 40 Apr 24 00:43:22 fir-md1-s1 kernel: 08 Apr 24 00:43:22 fir-md1-s1 kernel: 85 Apr 24 00:43:22 fir-md1-s1 kernel: c0 Apr 24 00:43:22 fir-md1-s1 kernel: 75 Apr 24 00:43:22 fir-md1-s1 kernel: 0f Apr 24 00:43:22 fir-md1-s1 kernel: 0f Apr 24 00:43:22 fir-md1-s1 kernel: 1f Apr 24 00:43:22 fir-md1-s1 kernel: 44 Apr 24 00:43:22 fir-md1-s1 kernel: 00 Apr 24 00:43:22 fir-md1-s1 kernel: 00 Apr 24 00:43:22 fir-md1-s1 kernel: f3 Apr 24 00:43:22 fir-md1-s1 kernel: 90 Apr 24 00:43:22 fir-md1-s1 kernel: <41> Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: 40 Apr 24 00:43:22 fir-md1-s1 kernel: 08 Apr 24 00:43:22 fir-md1-s1 kernel: 85 Apr 24 00:43:22 fir-md1-s1 kernel: c0 Apr 24 00:43:22 fir-md1-s1 kernel: 74 Apr 24 00:43:22 fir-md1-s1 kernel: f6 Apr 24 00:43:22 fir-md1-s1 kernel: 4d Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: 08 Apr 24 00:43:22 fir-md1-s1 kernel: 4d Apr 24 00:43:22 fir-md1-s1 kernel: 85 Apr 24 00:43:22 fir-md1-s1 kernel: c9 Apr 24 00:43:22 fir-md1-s1 kernel: 74 Apr 24 00:43:22 fir-md1-s1 kernel: 04 Apr 24 00:43:22 fir-md1-s1 kernel: 41 Apr 24 00:43:22 fir-md1-s1 kernel: 0f Apr 24 00:43:22 fir-md1-s1 kernel: 18 Apr 24 00:43:22 fir-md1-s1 kernel: 09 Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: Apr 24 00:43:22 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [mdt_io01_029:21699] Apr 24 00:43:22 fir-md1-s1 kernel: Modules linked in: Apr 24 00:43:22 fir-md1-s1 kernel: osp(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mdd(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lod(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mdt(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lfsck(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mgs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mgc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: osd_ldiskfs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lquota(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ldiskfs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lustre(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lmv(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mdc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: osc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lov(OE) Apr 24 00:43:22 fir-md1-s1 kernel: fid(OE) Apr 24 00:43:22 fir-md1-s1 kernel: fld(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ko2iblnd(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ptlrpc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: obdclass(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lnet(OE) Apr 24 00:43:22 fir-md1-s1 kernel: libcfs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: rpcsec_gss_krb5 Apr 24 00:43:22 fir-md1-s1 kernel: auth_rpcgss Apr 24 00:43:22 fir-md1-s1 kernel: nfsv4 Apr 24 00:43:22 fir-md1-s1 kernel: dns_resolver Apr 24 00:43:22 fir-md1-s1 kernel: nfs Apr 24 00:43:22 fir-md1-s1 kernel: lockd Apr 24 00:43:22 fir-md1-s1 kernel: grace Apr 24 00:43:22 fir-md1-s1 kernel: fscache Apr 24 00:43:22 fir-md1-s1 kernel: rdma_ucm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_ucm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: rdma_cm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: iw_cm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_ipoib(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_cm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_umad(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_fpga_tools(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_en(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_ib(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_core(OE) Apr 24 00:43:22 fir-md1-s1 kernel: dell_rbu Apr 24 00:43:22 fir-md1-s1 kernel: sunrpc Apr 24 00:43:22 fir-md1-s1 kernel: vfat Apr 24 00:43:22 fir-md1-s1 kernel: fat Apr 24 00:43:22 fir-md1-s1 kernel: dm_round_robin Apr 24 00:43:22 fir-md1-s1 kernel: amd64_edac_mod Apr 24 00:43:22 fir-md1-s1 kernel: edac_mce_amd Apr 24 00:43:22 fir-md1-s1 kernel: kvm_amd Apr 24 00:43:22 fir-md1-s1 kernel: kvm Apr 24 00:43:22 fir-md1-s1 kernel: irqbypass Apr 24 00:43:22 fir-md1-s1 kernel: crc32_pclmul Apr 24 00:43:22 fir-md1-s1 kernel: ghash_clmulni_intel Apr 24 00:43:22 fir-md1-s1 kernel: aesni_intel Apr 24 00:43:22 fir-md1-s1 kernel: lrw Apr 24 00:43:22 fir-md1-s1 kernel: gf128mul Apr 24 00:43:22 fir-md1-s1 kernel: glue_helper Apr 24 00:43:22 fir-md1-s1 kernel: ablk_helper Apr 24 00:43:22 fir-md1-s1 kernel: cryptd Apr 24 00:43:22 fir-md1-s1 kernel: dcdbas Apr 24 00:43:22 fir-md1-s1 kernel: ses Apr 24 00:43:22 fir-md1-s1 kernel: enclosure Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_si Apr 24 00:43:22 fir-md1-s1 kernel: pcspkr Apr 24 00:43:22 fir-md1-s1 kernel: dm_multipath Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_devintf Apr 24 00:43:22 fir-md1-s1 kernel: dm_mod Apr 24 00:43:22 fir-md1-s1 kernel: ccp Apr 24 00:43:22 fir-md1-s1 kernel: sg Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_msghandler Apr 24 00:43:22 fir-md1-s1 kernel: k10temp Apr 24 00:43:22 fir-md1-s1 kernel: i2c_piix4 Apr 24 00:43:22 fir-md1-s1 kernel: acpi_power_meter Apr 24 00:43:22 fir-md1-s1 kernel: knem(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ip_tables Apr 24 00:43:22 fir-md1-s1 kernel: ext4 Apr 24 00:43:22 fir-md1-s1 kernel: mbcache Apr 24 00:43:22 fir-md1-s1 kernel: jbd2 Apr 24 00:43:22 fir-md1-s1 kernel: sd_mod Apr 24 00:43:22 fir-md1-s1 kernel: crc_t10dif Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_generic Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_ib(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_uverbs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_core(OE) Apr 24 00:43:22 fir-md1-s1 kernel: i2c_algo_bit Apr 24 00:43:22 fir-md1-s1 kernel: drm_kms_helper Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_core(OE) Apr 24 00:43:22 fir-md1-s1 kernel: syscopyarea Apr 24 00:43:22 fir-md1-s1 kernel: mlxfw(OE) Apr 24 00:43:22 fir-md1-s1 kernel: sysfillrect Apr 24 00:43:22 fir-md1-s1 kernel: devlink Apr 24 00:43:22 fir-md1-s1 kernel: sysimgblt Apr 24 00:43:22 fir-md1-s1 kernel: fb_sys_fops Apr 24 00:43:22 fir-md1-s1 kernel: mlx_compat(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ahci Apr 24 00:43:22 fir-md1-s1 kernel: ttm Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_pclmul Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_common Apr 24 00:43:22 fir-md1-s1 kernel: libahci Apr 24 00:43:22 fir-md1-s1 kernel: drm Apr 24 00:43:22 fir-md1-s1 kernel: tg3 Apr 24 00:43:22 fir-md1-s1 kernel: crc32c_intel Apr 24 00:43:22 fir-md1-s1 kernel: ptp Apr 24 00:43:22 fir-md1-s1 kernel: libata Apr 24 00:43:22 fir-md1-s1 kernel: megaraid_sas Apr 24 00:43:22 fir-md1-s1 kernel: drm_panel_orientation_quirks Apr 24 00:43:22 fir-md1-s1 kernel: pps_core Apr 24 00:43:22 fir-md1-s1 kernel: mpt3sas(OE) Apr 24 00:43:22 fir-md1-s1 kernel: raid_class Apr 24 00:43:22 fir-md1-s1 kernel: scsi_transport_sas Apr 24 00:43:22 fir-md1-s1 kernel: Apr 24 00:43:22 fir-md1-s1 kernel: CPU: 9 PID: 21699 Comm: mdt_io01_029 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 Apr 24 00:43:22 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 Apr 24 00:43:22 fir-md1-s1 kernel: task: ffff8b7324381040 ti: ffff8b7324388000 task.ti: ffff8b7324388000 Apr 24 00:43:22 fir-md1-s1 kernel: RIP: 0010:[] Apr 24 00:43:22 fir-md1-s1 kernel: [] native_queued_spin_lock_slowpath+0x122/0x200 Apr 24 00:43:22 fir-md1-s1 kernel: RSP: 0018:ffff8b732438b800 EFLAGS: 00000246 Apr 24 00:43:22 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fb670ff0 RCX: 0000000000490000 Apr 24 00:43:22 fir-md1-s1 kernel: RDX: ffff8b533f61b780 RSI: 0000000000090101 RDI: ffff8b7339b45480 Apr 24 00:43:22 fir-md1-s1 kernel: RBP: ffff8b732438b800 R08: ffff8b533f69b780 R09: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: R10: ffff8b533f69f140 R11: fffff657e292ac00 R12: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: R13: ffff8b732438b7a0 R14: ffff8b48fb670d60 R15: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: FS: 00007f43a2d5c700(0000) GS:ffff8b533f680000(0000) knlGS:0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 24 00:43:22 fir-md1-s1 kernel: CR2: 00007f43a2dcf000 CR3: 0000000dd2c10000 CR4: 00000000003407e0 Apr 24 00:43:22 fir-md1-s1 kernel: Call Trace: Apr 24 00:43:22 fir-md1-s1 kernel: [] queued_spin_lock_slowpath+0xb/0xf Apr 24 00:43:22 fir-md1-s1 kernel: [] _raw_spin_lock+0x20/0x30 Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ktime_get_ts64+0x52/0xf0 Apr 24 00:43:22 fir-md1-s1 kernel: [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] mdt_obd_preprw+0x637/0x1060 [mdt] Apr 24 00:43:22 fir-md1-s1 kernel: [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? update_curr+0x14c/0x1e0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? account_entity_dequeue+0xae/0xd0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? default_wake_function+0x12/0x20 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? __wake_up_common+0x5b/0x90 Apr 24 00:43:22 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:22 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:22 fir-md1-s1 kernel: Code: Apr 24 00:43:22 fir-md1-s1 kernel: 13 Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: c1 Apr 24 00:43:22 fir-md1-s1 kernel: ea Apr 24 00:43:22 fir-md1-s1 kernel: 0d Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: 98 Apr 24 00:43:22 fir-md1-s1 kernel: 83 Apr 24 00:43:22 fir-md1-s1 kernel: e2 Apr 24 00:43:22 fir-md1-s1 kernel: 30 Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: 81 Apr 24 00:43:22 fir-md1-s1 kernel: c2 Apr 24 00:43:22 fir-md1-s1 kernel: 80 Apr 24 00:43:22 fir-md1-s1 kernel: b7 Apr 24 00:43:22 fir-md1-s1 kernel: 01 Apr 24 00:43:22 fir-md1-s1 kernel: 00 Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: 03 Apr 24 00:43:22 fir-md1-s1 kernel: 14 Apr 24 00:43:22 fir-md1-s1 kernel: c5 Apr 24 00:43:22 fir-md1-s1 kernel: 60 Apr 24 00:43:22 fir-md1-s1 kernel: b9 Apr 24 00:43:22 fir-md1-s1 kernel: 14 Apr 24 00:43:22 fir-md1-s1 kernel: 9c Apr 24 00:43:22 fir-md1-s1 kernel: 4c Apr 24 00:43:22 fir-md1-s1 kernel: 89 Apr 24 00:43:22 fir-md1-s1 kernel: 02 Apr 24 00:43:22 fir-md1-s1 kernel: 41 Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: 40 Apr 24 00:43:22 fir-md1-s1 kernel: 08 Apr 24 00:43:22 fir-md1-s1 kernel: 85 Apr 24 00:43:22 fir-md1-s1 kernel: c0 Apr 24 00:43:22 fir-md1-s1 kernel: 75 Apr 24 00:43:22 fir-md1-s1 kernel: 0f Apr 24 00:43:22 fir-md1-s1 kernel: 0f Apr 24 00:43:22 fir-md1-s1 kernel: 1f Apr 24 00:43:22 fir-md1-s1 kernel: 44 Apr 24 00:43:22 fir-md1-s1 kernel: 00 Apr 24 00:43:22 fir-md1-s1 kernel: 00 Apr 24 00:43:22 fir-md1-s1 kernel: f3 Apr 24 00:43:22 fir-md1-s1 kernel: 90 Apr 24 00:43:22 fir-md1-s1 kernel: <41> Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: 40 Apr 24 00:43:22 fir-md1-s1 kernel: 08 Apr 24 00:43:22 fir-md1-s1 kernel: 85 Apr 24 00:43:22 fir-md1-s1 kernel: c0 Apr 24 00:43:22 fir-md1-s1 kernel: 74 Apr 24 00:43:22 fir-md1-s1 kernel: f6 Apr 24 00:43:22 fir-md1-s1 kernel: 4d Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: 08 Apr 24 00:43:22 fir-md1-s1 kernel: 4d Apr 24 00:43:22 fir-md1-s1 kernel: 85 Apr 24 00:43:22 fir-md1-s1 kernel: c9 Apr 24 00:43:22 fir-md1-s1 kernel: 74 Apr 24 00:43:22 fir-md1-s1 kernel: 04 Apr 24 00:43:22 fir-md1-s1 kernel: 41 Apr 24 00:43:22 fir-md1-s1 kernel: 0f Apr 24 00:43:22 fir-md1-s1 kernel: 18 Apr 24 00:43:22 fir-md1-s1 kernel: 09 Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: Apr 24 00:43:22 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [mdt_io02_030:21634] Apr 24 00:43:22 fir-md1-s1 kernel: Modules linked in: Apr 24 00:43:22 fir-md1-s1 kernel: osp(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mdd(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lod(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mdt(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lfsck(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mgs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mgc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: osd_ldiskfs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lquota(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ldiskfs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lustre(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lmv(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mdc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: osc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lov(OE) Apr 24 00:43:22 fir-md1-s1 kernel: fid(OE) Apr 24 00:43:22 fir-md1-s1 kernel: fld(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ko2iblnd(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ptlrpc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: obdclass(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lnet(OE) Apr 24 00:43:22 fir-md1-s1 kernel: libcfs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: rpcsec_gss_krb5 Apr 24 00:43:22 fir-md1-s1 kernel: auth_rpcgss Apr 24 00:43:22 fir-md1-s1 kernel: nfsv4 Apr 24 00:43:22 fir-md1-s1 kernel: dns_resolver Apr 24 00:43:22 fir-md1-s1 kernel: nfs Apr 24 00:43:22 fir-md1-s1 kernel: lockd Apr 24 00:43:22 fir-md1-s1 kernel: grace Apr 24 00:43:22 fir-md1-s1 kernel: fscache Apr 24 00:43:22 fir-md1-s1 kernel: rdma_ucm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_ucm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: rdma_cm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: iw_cm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_ipoib(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_cm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_umad(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_fpga_tools(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_en(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_ib(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_core(OE) Apr 24 00:43:22 fir-md1-s1 kernel: dell_rbu Apr 24 00:43:22 fir-md1-s1 kernel: sunrpc Apr 24 00:43:22 fir-md1-s1 kernel: vfat Apr 24 00:43:22 fir-md1-s1 kernel: fat Apr 24 00:43:22 fir-md1-s1 kernel: dm_round_robin Apr 24 00:43:22 fir-md1-s1 kernel: amd64_edac_mod Apr 24 00:43:22 fir-md1-s1 kernel: edac_mce_amd Apr 24 00:43:22 fir-md1-s1 kernel: kvm_amd Apr 24 00:43:22 fir-md1-s1 kernel: kvm Apr 24 00:43:22 fir-md1-s1 kernel: irqbypass Apr 24 00:43:22 fir-md1-s1 kernel: crc32_pclmul Apr 24 00:43:22 fir-md1-s1 kernel: ghash_clmulni_intel Apr 24 00:43:22 fir-md1-s1 kernel: aesni_intel Apr 24 00:43:22 fir-md1-s1 kernel: lrw Apr 24 00:43:22 fir-md1-s1 kernel: gf128mul Apr 24 00:43:22 fir-md1-s1 kernel: glue_helper Apr 24 00:43:22 fir-md1-s1 kernel: ablk_helper Apr 24 00:43:22 fir-md1-s1 kernel: cryptd Apr 24 00:43:22 fir-md1-s1 kernel: dcdbas Apr 24 00:43:22 fir-md1-s1 kernel: ses Apr 24 00:43:22 fir-md1-s1 kernel: enclosure Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_si Apr 24 00:43:22 fir-md1-s1 kernel: pcspkr Apr 24 00:43:22 fir-md1-s1 kernel: dm_multipath Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_devintf Apr 24 00:43:22 fir-md1-s1 kernel: dm_mod Apr 24 00:43:22 fir-md1-s1 kernel: ccp Apr 24 00:43:22 fir-md1-s1 kernel: sg Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_msghandler Apr 24 00:43:22 fir-md1-s1 kernel: k10temp Apr 24 00:43:22 fir-md1-s1 kernel: i2c_piix4 Apr 24 00:43:22 fir-md1-s1 kernel: acpi_power_meter Apr 24 00:43:22 fir-md1-s1 kernel: knem(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ip_tables Apr 24 00:43:22 fir-md1-s1 kernel: ext4 Apr 24 00:43:22 fir-md1-s1 kernel: mbcache Apr 24 00:43:22 fir-md1-s1 kernel: jbd2 Apr 24 00:43:22 fir-md1-s1 kernel: sd_mod Apr 24 00:43:22 fir-md1-s1 kernel: crc_t10dif Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_generic Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_ib(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_uverbs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_core(OE) Apr 24 00:43:22 fir-md1-s1 kernel: i2c_algo_bit Apr 24 00:43:22 fir-md1-s1 kernel: drm_kms_helper Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_core(OE) Apr 24 00:43:22 fir-md1-s1 kernel: syscopyarea Apr 24 00:43:22 fir-md1-s1 kernel: mlxfw(OE) Apr 24 00:43:22 fir-md1-s1 kernel: sysfillrect Apr 24 00:43:22 fir-md1-s1 kernel: devlink Apr 24 00:43:22 fir-md1-s1 kernel: sysimgblt Apr 24 00:43:22 fir-md1-s1 kernel: fb_sys_fops Apr 24 00:43:22 fir-md1-s1 kernel: mlx_compat(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ahci Apr 24 00:43:22 fir-md1-s1 kernel: ttm Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_pclmul Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_common Apr 24 00:43:22 fir-md1-s1 kernel: libahci Apr 24 00:43:22 fir-md1-s1 kernel: drm Apr 24 00:43:22 fir-md1-s1 kernel: tg3 Apr 24 00:43:22 fir-md1-s1 kernel: crc32c_intel Apr 24 00:43:22 fir-md1-s1 kernel: ptp Apr 24 00:43:22 fir-md1-s1 kernel: libata Apr 24 00:43:22 fir-md1-s1 kernel: megaraid_sas Apr 24 00:43:22 fir-md1-s1 kernel: drm_panel_orientation_quirks Apr 24 00:43:22 fir-md1-s1 kernel: pps_core Apr 24 00:43:22 fir-md1-s1 kernel: mpt3sas(OE) Apr 24 00:43:22 fir-md1-s1 kernel: raid_class Apr 24 00:43:22 fir-md1-s1 kernel: scsi_transport_sas Apr 24 00:43:22 fir-md1-s1 kernel: Apr 24 00:43:22 fir-md1-s1 kernel: CPU: 10 PID: 21634 Comm: mdt_io02_030 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 Apr 24 00:43:22 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 Apr 24 00:43:22 fir-md1-s1 kernel: task: ffff8b7328f86180 ti: ffff8b732256c000 task.ti: ffff8b732256c000 Apr 24 00:43:22 fir-md1-s1 kernel: RIP: 0010:[] Apr 24 00:43:22 fir-md1-s1 kernel: [] native_queued_spin_lock_slowpath+0x122/0x200 Apr 24 00:43:22 fir-md1-s1 kernel: RSP: 0018:ffff8b732256f800 EFLAGS: 00000246 Apr 24 00:43:22 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b5d5fe93da8 RCX: 0000000000510000 Apr 24 00:43:22 fir-md1-s1 kernel: RDX: ffff8b433f05b780 RSI: 0000000001210101 RDI: ffff8b7339b45480 Apr 24 00:43:22 fir-md1-s1 kernel: RBP: ffff8b732256f800 R08: ffff8b633f69b780 R09: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: R10: ffff8b633f69f140 R11: fffff6582b75a000 R12: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: R13: ffff8b732256f7a0 R14: ffff8b5d5fe93b18 R15: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: FS: 00007fadc53ed700(0000) GS:ffff8b633f680000(0000) knlGS:0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 24 00:43:22 fir-md1-s1 kernel: CR2: 00007fadc7f4d000 CR3: 0000000dd2c10000 CR4: 00000000003407e0 Apr 24 00:43:22 fir-md1-s1 kernel: Call Trace: Apr 24 00:43:22 fir-md1-s1 kernel: [] queued_spin_lock_slowpath+0xb/0xf Apr 24 00:43:22 fir-md1-s1 kernel: [] _raw_spin_lock+0x20/0x30 Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? vsnprintf+0x234/0x6a0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ktime_get_ts64+0x52/0xf0 Apr 24 00:43:22 fir-md1-s1 kernel: [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] mdt_obd_preprw+0x637/0x1060 [mdt] Apr 24 00:43:22 fir-md1-s1 kernel: [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? update_curr+0x14c/0x1e0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? account_entity_dequeue+0xae/0xd0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? default_wake_function+0x12/0x20 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? __wake_up_common+0x5b/0x90 Apr 24 00:43:22 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:22 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:22 fir-md1-s1 kernel: Code: Apr 24 00:43:22 fir-md1-s1 kernel: 13 Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: c1 Apr 24 00:43:22 fir-md1-s1 kernel: ea Apr 24 00:43:22 fir-md1-s1 kernel: 0d Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: 98 Apr 24 00:43:22 fir-md1-s1 kernel: 83 Apr 24 00:43:22 fir-md1-s1 kernel: e2 Apr 24 00:43:22 fir-md1-s1 kernel: 30 Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: 81 Apr 24 00:43:22 fir-md1-s1 kernel: c2 Apr 24 00:43:22 fir-md1-s1 kernel: 80 Apr 24 00:43:22 fir-md1-s1 kernel: b7 Apr 24 00:43:22 fir-md1-s1 kernel: 01 Apr 24 00:43:22 fir-md1-s1 kernel: 00 Apr 24 00:43:22 fir-md1-s1 kernel: 48 Apr 24 00:43:22 fir-md1-s1 kernel: 03 Apr 24 00:43:22 fir-md1-s1 kernel: 14 Apr 24 00:43:22 fir-md1-s1 kernel: c5 Apr 24 00:43:22 fir-md1-s1 kernel: 60 Apr 24 00:43:22 fir-md1-s1 kernel: b9 Apr 24 00:43:22 fir-md1-s1 kernel: 14 Apr 24 00:43:22 fir-md1-s1 kernel: 9c Apr 24 00:43:22 fir-md1-s1 kernel: 4c Apr 24 00:43:22 fir-md1-s1 kernel: 89 Apr 24 00:43:22 fir-md1-s1 kernel: 02 Apr 24 00:43:22 fir-md1-s1 kernel: 41 Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: 40 Apr 24 00:43:22 fir-md1-s1 kernel: 08 Apr 24 00:43:22 fir-md1-s1 kernel: 85 Apr 24 00:43:22 fir-md1-s1 kernel: c0 Apr 24 00:43:22 fir-md1-s1 kernel: 75 Apr 24 00:43:22 fir-md1-s1 kernel: 0f Apr 24 00:43:22 fir-md1-s1 kernel: 0f Apr 24 00:43:22 fir-md1-s1 kernel: 1f Apr 24 00:43:22 fir-md1-s1 kernel: 44 Apr 24 00:43:22 fir-md1-s1 kernel: 00 Apr 24 00:43:22 fir-md1-s1 kernel: 00 Apr 24 00:43:22 fir-md1-s1 kernel: f3 Apr 24 00:43:22 fir-md1-s1 kernel: 90 Apr 24 00:43:22 fir-md1-s1 kernel: <41> Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: 40 Apr 24 00:43:22 fir-md1-s1 kernel: 08 Apr 24 00:43:22 fir-md1-s1 kernel: 85 Apr 24 00:43:22 fir-md1-s1 kernel: c0 Apr 24 00:43:22 fir-md1-s1 kernel: 74 Apr 24 00:43:22 fir-md1-s1 kernel: f6 Apr 24 00:43:22 fir-md1-s1 kernel: 4d Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: 08 Apr 24 00:43:22 fir-md1-s1 kernel: 4d Apr 24 00:43:22 fir-md1-s1 kernel: 85 Apr 24 00:43:22 fir-md1-s1 kernel: c9 Apr 24 00:43:22 fir-md1-s1 kernel: 74 Apr 24 00:43:22 fir-md1-s1 kernel: 04 Apr 24 00:43:22 fir-md1-s1 kernel: 41 Apr 24 00:43:22 fir-md1-s1 kernel: 0f Apr 24 00:43:22 fir-md1-s1 kernel: 18 Apr 24 00:43:22 fir-md1-s1 kernel: 09 Apr 24 00:43:22 fir-md1-s1 kernel: 8b Apr 24 00:43:22 fir-md1-s1 kernel: Apr 24 00:43:22 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [mdt_io01_001:20782] Apr 24 00:43:22 fir-md1-s1 kernel: Modules linked in: Apr 24 00:43:22 fir-md1-s1 kernel: osp(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mdd(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lod(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mdt(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lfsck(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mgs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mgc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: osd_ldiskfs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lquota(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ldiskfs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lustre(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lmv(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mdc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: osc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lov(OE) Apr 24 00:43:22 fir-md1-s1 kernel: fid(OE) Apr 24 00:43:22 fir-md1-s1 kernel: fld(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ko2iblnd(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ptlrpc(OE) Apr 24 00:43:22 fir-md1-s1 kernel: obdclass(OE) Apr 24 00:43:22 fir-md1-s1 kernel: lnet(OE) Apr 24 00:43:22 fir-md1-s1 kernel: libcfs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: rpcsec_gss_krb5 Apr 24 00:43:22 fir-md1-s1 kernel: auth_rpcgss Apr 24 00:43:22 fir-md1-s1 kernel: nfsv4 Apr 24 00:43:22 fir-md1-s1 kernel: dns_resolver Apr 24 00:43:22 fir-md1-s1 kernel: nfs Apr 24 00:43:22 fir-md1-s1 kernel: lockd Apr 24 00:43:22 fir-md1-s1 kernel: grace Apr 24 00:43:22 fir-md1-s1 kernel: fscache Apr 24 00:43:22 fir-md1-s1 kernel: rdma_ucm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_ucm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: rdma_cm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: iw_cm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_ipoib(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_cm(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_umad(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_fpga_tools(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_en(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_ib(OE) Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_core(OE) Apr 24 00:43:22 fir-md1-s1 kernel: dell_rbu Apr 24 00:43:22 fir-md1-s1 kernel: sunrpc Apr 24 00:43:22 fir-md1-s1 kernel: vfat Apr 24 00:43:22 fir-md1-s1 kernel: fat Apr 24 00:43:22 fir-md1-s1 kernel: dm_round_robin Apr 24 00:43:22 fir-md1-s1 kernel: amd64_edac_mod Apr 24 00:43:22 fir-md1-s1 kernel: edac_mce_amd Apr 24 00:43:22 fir-md1-s1 kernel: kvm_amd Apr 24 00:43:22 fir-md1-s1 kernel: kvm Apr 24 00:43:22 fir-md1-s1 kernel: irqbypass Apr 24 00:43:22 fir-md1-s1 kernel: crc32_pclmul Apr 24 00:43:22 fir-md1-s1 kernel: ghash_clmulni_intel Apr 24 00:43:22 fir-md1-s1 kernel: aesni_intel Apr 24 00:43:22 fir-md1-s1 kernel: lrw Apr 24 00:43:22 fir-md1-s1 kernel: gf128mul Apr 24 00:43:22 fir-md1-s1 kernel: glue_helper Apr 24 00:43:22 fir-md1-s1 kernel: ablk_helper Apr 24 00:43:22 fir-md1-s1 kernel: cryptd Apr 24 00:43:22 fir-md1-s1 kernel: dcdbas Apr 24 00:43:22 fir-md1-s1 kernel: ses Apr 24 00:43:22 fir-md1-s1 kernel: enclosure Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_si Apr 24 00:43:22 fir-md1-s1 kernel: pcspkr Apr 24 00:43:22 fir-md1-s1 kernel: dm_multipath Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_devintf Apr 24 00:43:22 fir-md1-s1 kernel: dm_mod Apr 24 00:43:22 fir-md1-s1 kernel: ccp Apr 24 00:43:22 fir-md1-s1 kernel: sg Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_msghandler Apr 24 00:43:22 fir-md1-s1 kernel: k10temp Apr 24 00:43:22 fir-md1-s1 kernel: i2c_piix4 Apr 24 00:43:22 fir-md1-s1 kernel: acpi_power_meter Apr 24 00:43:22 fir-md1-s1 kernel: knem(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ip_tables Apr 24 00:43:22 fir-md1-s1 kernel: ext4 Apr 24 00:43:22 fir-md1-s1 kernel: mbcache Apr 24 00:43:22 fir-md1-s1 kernel: jbd2 Apr 24 00:43:22 fir-md1-s1 kernel: sd_mod Apr 24 00:43:22 fir-md1-s1 kernel: crc_t10dif Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_generic Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_ib(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_uverbs(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ib_core(OE) Apr 24 00:43:22 fir-md1-s1 kernel: i2c_algo_bit Apr 24 00:43:22 fir-md1-s1 kernel: drm_kms_helper Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_core(OE) Apr 24 00:43:22 fir-md1-s1 kernel: syscopyarea Apr 24 00:43:22 fir-md1-s1 kernel: mlxfw(OE) Apr 24 00:43:22 fir-md1-s1 kernel: sysfillrect Apr 24 00:43:22 fir-md1-s1 kernel: devlink Apr 24 00:43:22 fir-md1-s1 kernel: sysimgblt Apr 24 00:43:22 fir-md1-s1 kernel: fb_sys_fops Apr 24 00:43:22 fir-md1-s1 kernel: mlx_compat(OE) Apr 24 00:43:22 fir-md1-s1 kernel: ahci Apr 24 00:43:22 fir-md1-s1 kernel: ttm Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_pclmul Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_common Apr 24 00:43:22 fir-md1-s1 kernel: libahci Apr 24 00:43:22 fir-md1-s1 kernel: drm Apr 24 00:43:22 fir-md1-s1 kernel: tg3 Apr 24 00:43:22 fir-md1-s1 kernel: crc32c_intel Apr 24 00:43:22 fir-md1-s1 kernel: ptp Apr 24 00:43:22 fir-md1-s1 kernel: libata Apr 24 00:43:22 fir-md1-s1 kernel: megaraid_sas Apr 24 00:43:22 fir-md1-s1 kernel: drm_panel_orientation_quirks Apr 24 00:43:22 fir-md1-s1 kernel: pps_core Apr 24 00:43:22 fir-md1-s1 kernel: mpt3sas(OE) Apr 24 00:43:22 fir-md1-s1 kernel: raid_class Apr 24 00:43:22 fir-md1-s1 kernel: scsi_transport_sas Apr 24 00:43:22 fir-md1-s1 kernel: Apr 24 00:43:22 fir-md1-s1 kernel: CPU: 13 PID: 20782 Comm: mdt_io01_001 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 Apr 24 00:43:22 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 Apr 24 00:43:22 fir-md1-s1 kernel: task: ffff8b7330309040 ti: ffff8b73392ac000 task.ti: ffff8b73392ac000 Apr 24 00:43:22 fir-md1-s1 kernel: RIP: 0010:[] Apr 24 00:43:22 fir-md1-s1 kernel: [] native_queued_spin_lock_slowpath+0x122/0x200 Apr 24 00:43:22 fir-md1-s1 kernel: RSP: 0018:ffff8b73392af800 EFLAGS: 00000246 Apr 24 00:43:22 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fb62c1d0 RCX: 0000000000690000 Apr 24 00:43:22 fir-md1-s1 kernel: RDX: ffff8b433ee5b780 RSI: 0000000000210101 RDI: ffff8b7339b45480 Apr 24 00:43:22 fir-md1-s1 kernel: RBP: ffff8b73392af800 R08: ffff8b533f6db780 R09: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: R10: ffff8b533f6df140 R11: fffff657da9eda00 R12: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: R13: ffff8b73392af7a0 R14: ffff8b48fb62bf40 R15: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: FS: 00007f1966f48880(0000) GS:ffff8b533f6c0000(0000) knlGS:0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 24 00:43:22 fir-md1-s1 kernel: CR2: 00007f1530508764 CR3: 0000002031f6e000 CR4: 00000000003407e0 Apr 24 00:43:22 fir-md1-s1 kernel: Call Trace: Apr 24 00:43:22 fir-md1-s1 kernel: [] queued_spin_lock_slowpath+0xb/0xf Apr 24 00:43:22 fir-md1-s1 kernel: [] _raw_spin_lock+0x20/0x30 Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ktime_get_ts64+0x52/0xf0 Apr 24 00:43:22 fir-md1-s1 kernel: [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] mdt_obd_preprw+0x637/0x1060 [mdt] Apr 24 00:43:22 fir-md1-s1 kernel: [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? update_curr+0x14c/0x1e0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? account_entity_dequeue+0xae/0xd0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas Apr 24 00:43:22 fir-md1-s1 kernel: CPU: 0 PID: 21591 Comm: mdt00_039 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 Apr 24 00:43:22 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 Apr 24 00:43:22 fir-md1-s1 kernel: task: ffff8b7322485140 ti: ffff8b7324208000 task.ti: ffff8b7324208000 Apr 24 00:43:22 fir-md1-s1 kernel: RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x156/0x200 Apr 24 00:43:22 fir-md1-s1 kernel: RSP: 0018:ffff8b732420b508 EFLAGS: 00000202 Apr 24 00:43:22 fir-md1-s1 kernel: RAX: 0000000000000101 RBX: ffff8b3793b82db8 RCX: 0000000000010000 Apr 24 00:43:22 fir-md1-s1 kernel: RDX: 0000000000410101 RSI: 0000000000000101 RDI: ffff8b7339b45480 Apr 24 00:43:22 fir-md1-s1 kernel: RBP: ffff8b732420b508 R08: ffff8b433ee1b780 R09: 0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: R10: ffff8b433ee1f140 R11: fffff6579bdefa00 R12: ffff8b732420b4a8 Apr 24 00:43:22 fir-md1-s1 kernel: R13: ffff8b3793b82ea0 R14: 0000000000000000 R15: 1fffffffffffffff Apr 24 00:43:22 fir-md1-s1 kernel: FS: 00007f73d2277880(0000) GS:ffff8b433ee00000(0000) knlGS:0000000000000000 Apr 24 00:43:22 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 24 00:43:22 fir-md1-s1 kernel: CR2: 0000000000485110 CR3: 0000001038288000 CR4: 00000000003407f0 Apr 24 00:43:22 fir-md1-s1 kernel: Call Trace: Apr 24 00:43:22 fir-md1-s1 kernel: [] queued_spin_lock_slowpath+0xb/0xf Apr 24 00:43:22 fir-md1-s1 kernel: [] _raw_spin_lock+0x20/0x30 Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? lprocfs_counter_sub+0xc1/0x130 [obdclass] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? dynlock_unlock+0x194/0x1e0 [osd_ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? __brelse+0x3d/0x50 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? iam_path_release+0x42/0x60 [osd_ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_getblk+0x65/0x200 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_bread+0x27/0xc0 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_append+0x81/0x150 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_init_new_dir+0xcf/0x230 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? __brelse+0x3d/0x50 Apr 24 00:43:22 fir-md1-s1 kernel: [] ldiskfs_add_dot_dotdot+0x4e/0x90 [ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] osd_add_dot_dotdot_internal.isra.77+0x5f/0x80 [osd_ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] osd_index_ea_insert+0xb1a/0x1240 [osd_ldiskfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] lod_sub_insert+0x1c1/0x340 [lod] Apr 24 00:43:22 fir-md1-s1 kernel: [] lod_insert+0x24/0x30 [lod] Apr 24 00:43:22 fir-md1-s1 kernel: [] __mdd_index_insert_only+0x1cc/0x280 [mdd] Apr 24 00:43:22 fir-md1-s1 kernel: [] mdd_create_object+0x6c8/0x820 [mdd] Apr 24 00:43:22 fir-md1-s1 kernel: [] mdd_create+0xd80/0x1440 [mdd] Apr 24 00:43:22 fir-md1-s1 kernel: [] mdt_create+0xb54/0x1090 [mdt] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? lprocfs_job_stats_log+0xd1/0x640 [obdclass] Apr 24 00:43:22 fir-md1-s1 kernel: [] mdt_reint_create+0x16b/0x360 [mdt] Apr 24 00:43:22 fir-md1-s1 kernel: [] mdt_reint_rec+0x83/0x210 [mdt] Apr 24 00:43:22 fir-md1-s1 kernel: [] mdt_reint_internal+0x6e3/0xaf0 [mdt] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? mdt_thread_info_init+0xa4/0x1e0 [mdt] Apr 24 00:43:22 fir-md1-s1 kernel: [] mdt_reint+0x67/0x140 [mdt] Apr 24 00:43:22 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:22 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? default_wake_function+0x12/0x20 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? __wake_up_common+0x5b/0x90 Apr 24 00:43:22 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Apr 24 00:43:22 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:22 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Apr 24 00:43:22 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:22 fir-md1-s1 kernel: Code: 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 <8b> 17 0f b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84 Apr 24 00:43:27 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [mdt_io01_016:21594] Apr 24 00:43:27 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif Apr 24 00:43:27 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas Apr 24 00:43:27 fir-md1-s1 kernel: CPU: 17 PID: 21594 Comm: mdt_io01_016 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 Apr 24 00:43:27 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 Apr 24 00:43:27 fir-md1-s1 kernel: task: ffff8b733ecd1040 ti: ffff8b733ecdc000 task.ti: ffff8b733ecdc000 Apr 24 00:43:27 fir-md1-s1 kernel: RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x1ce/0x200 Apr 24 00:43:27 fir-md1-s1 kernel: RSP: 0018:ffff8b733ecdf790 EFLAGS: 00000202 Apr 24 00:43:27 fir-md1-s1 kernel: RAX: 0000000000000001 RBX: ffff8b733ecdf780 RCX: 0000000000000001 Apr 24 00:43:27 fir-md1-s1 kernel: RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff8b7339b45480 Apr 24 00:43:27 fir-md1-s1 kernel: RBP: ffff8b733ecdf790 R08: 0000000000000101 R09: ffffffffc1204d1a Apr 24 00:43:27 fir-md1-s1 kernel: R10: ffff8b533f71f140 R11: fffff657dd4db400 R12: 0000005300000000 Apr 24 00:43:27 fir-md1-s1 kernel: R13: ffff8b733ecdf70e R14: 0000000a0000ffff R15: ffffffff9b781153 Apr 24 00:43:27 fir-md1-s1 kernel: FS: 00007f43a2d5c700(0000) GS:ffff8b533f700000(0000) knlGS:0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 24 00:43:27 fir-md1-s1 kernel: CR2: 00007f43a2dcf000 CR3: 0000000dd2c10000 CR4: 00000000003407e0 Apr 24 00:43:27 fir-md1-s1 kernel: Call Trace: Apr 24 00:43:27 fir-md1-s1 kernel: [] queued_spin_lock_slowpath+0xb/0xf Apr 24 00:43:27 fir-md1-s1 kernel: [] _raw_spin_lock+0x20/0x30 Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib] Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] Apr 24 00:43:27 fir-md1-s1 kernel: [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] osd_read_prep+0x2de/0x400 [osd_ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] mdt_obd_preprw+0xd5f/0x1060 [mdt] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? put_dec+0x72/0x90 Apr 24 00:43:27 fir-md1-s1 kernel: [] tgt_brw_read+0x9db/0x1e50 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? vsnprintf+0x234/0x6a0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? null_alloc_rs+0x186/0x340 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? default_wake_function+0x12/0x20 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? __wake_up_common+0x5b/0x90 Apr 24 00:43:27 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:27 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:27 fir-md1-s1 kernel: Code: 37 81 fe 00 01 00 00 74 f4 e9 93 fe ff ff 0f 1f 80 00 00 00 00 83 fa 01 75 11 0f 1f 00 e9 68 fe ff ff 0f 1f 00 85 c0 74 0c f3 90 <8b> 07 0f b6 c0 83 f8 03 75 f0 b8 01 00 00 00 66 89 07 5d c3 66 Apr 24 00:43:27 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [mdt_io02_028:21630] Apr 24 00:43:27 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif Apr 24 00:43:27 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas Apr 24 00:43:27 fir-md1-s1 kernel: CPU: 30 PID: 21630 Comm: mdt_io02_028 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 Apr 24 00:43:27 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 Apr 24 00:43:27 fir-md1-s1 kernel: task: ffff8b7328f82080 ti: ffff8b7321174000 task.ti: ffff8b7321174000 Apr 24 00:43:27 fir-md1-s1 kernel: RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 Apr 24 00:43:27 fir-md1-s1 kernel: RSP: 0018:ffff8b7321177800 EFLAGS: 00000246 Apr 24 00:43:27 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fb62b980 RCX: 0000000000f10000 Apr 24 00:43:27 fir-md1-s1 kernel: RDX: ffff8b533f81b780 RSI: 0000000001090101 RDI: ffff8b7339b45480 Apr 24 00:43:27 fir-md1-s1 kernel: RBP: ffff8b7321177800 R08: ffff8b633f7db780 R09: 0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: R10: ffff8b633f7df140 R11: fffff658360b8e00 R12: 0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: R13: ffff8b73211777a0 R14: ffff8b48fb62b6f0 R15: 0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: FS: 00007f35da56d880(0000) GS:ffff8b633f7c0000(0000) knlGS:0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 24 00:43:27 fir-md1-s1 kernel: CR2: 00007f35c9809ff0 CR3: 0000000dd2c10000 CR4: 00000000003407e0 Apr 24 00:43:27 fir-md1-s1 kernel: Call Trace: Apr 24 00:43:27 fir-md1-s1 kernel: [] queued_spin_lock_slowpath+0xb/0xf Apr 24 00:43:27 fir-md1-s1 kernel: [] _raw_spin_lock+0x20/0x30 Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ktime_get_ts64+0x52/0xf0 Apr 24 00:43:27 fir-md1-s1 kernel: [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] mdt_obd_preprw+0x637/0x1060 [mdt] Apr 24 00:43:27 fir-md1-s1 kernel: [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? default_wake_function+0x12/0x20 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? __wake_up_common+0x5b/0x90 Apr 24 00:43:27 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:27 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:27 fir-md1-s1 kernel: Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 14 9c 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b Apr 24 00:43:27 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#33 stuck for 22s! [mdt_io01_111:22207] Apr 24 00:43:27 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif Apr 24 00:43:27 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas Apr 24 00:43:27 fir-md1-s1 kernel: CPU: 33 PID: 22207 Comm: mdt_io01_111 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 Apr 24 00:43:27 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 Apr 24 00:43:27 fir-md1-s1 kernel: task: ffff8b7322a74100 ti: ffff8b7322a9c000 task.ti: ffff8b7322a9c000 Apr 24 00:43:27 fir-md1-s1 kernel: RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x126/0x200 Apr 24 00:43:27 fir-md1-s1 kernel: RSP: 0018:ffff8b7322a9f800 EFLAGS: 00000246 Apr 24 00:43:27 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fb62bda8 RCX: 0000000001090000 Apr 24 00:43:27 fir-md1-s1 kernel: RDX: ffff8b533f6db780 RSI: 0000000000690101 RDI: ffff8b7339b45480 Apr 24 00:43:27 fir-md1-s1 kernel: RBP: ffff8b7322a9f800 R08: ffff8b533f81b780 R09: 0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: R10: ffff8b533f81f140 R11: fffff657d8defe00 R12: 0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: R13: ffff8b7322a9f7a0 R14: ffff8b48fb62bb18 R15: 0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: FS: 00007fa109a46740(0000) GS:ffff8b533f800000(0000) knlGS:0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 24 00:43:27 fir-md1-s1 kernel: CR2: 00007fa1096331cc CR3: 0000000dd2c10000 CR4: 00000000003407e0 Apr 24 00:43:27 fir-md1-s1 kernel: Call Trace: Apr 24 00:43:27 fir-md1-s1 kernel: [] queued_spin_lock_slowpath+0xb/0xf Apr 24 00:43:27 fir-md1-s1 kernel: [] _raw_spin_lock+0x20/0x30 Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ktime_get_ts64+0x52/0xf0 Apr 24 00:43:27 fir-md1-s1 kernel: [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] mdt_obd_preprw+0x637/0x1060 [mdt] Apr 24 00:43:27 fir-md1-s1 kernel: [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? update_curr+0x14c/0x1e0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? account_entity_dequeue+0xae/0xd0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? default_wake_function+0x12/0x20 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? __wake_up_common+0x5b/0x90 Apr 24 00:43:27 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:27 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:27 fir-md1-s1 kernel: Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 14 9c 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 Apr 24 00:43:27 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#36 stuck for 22s! [mdt_io00_018:21579] Apr 24 00:43:27 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif Apr 24 00:43:27 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas Apr 24 00:43:27 fir-md1-s1 kernel: CPU: 36 PID: 21579 Comm: mdt_io00_018 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 Apr 24 00:43:27 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 Apr 24 00:43:27 fir-md1-s1 kernel: task: ffff8b733f0ce180 ti: ffff8b7322144000 task.ti: ffff8b7322144000 Apr 24 00:43:27 fir-md1-s1 kernel: RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x128/0x200 Apr 24 00:43:27 fir-md1-s1 kernel: RSP: 0018:ffff8b7322147750 EFLAGS: 00000246 Apr 24 00:43:27 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b5d6cc4f3b0 RCX: 0000000001210000 Apr 24 00:43:27 fir-md1-s1 kernel: RDX: ffff8b433ee1b780 RSI: 0000000000010101 RDI: ffff8b7339b45480 Apr 24 00:43:27 fir-md1-s1 kernel: RBP: ffff8b7322147750 R08: ffff8b433f05b780 R09: 0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: R10: ffff8b433f05f140 R11: fffff6579cf19600 R12: 0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: R13: ffff8b73221476f0 R14: ffff8b5d6cc4f120 R15: 0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: FS: 00007f0c659f7740(0000) GS:ffff8b433f040000(0000) knlGS:0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 24 00:43:27 fir-md1-s1 kernel: CR2: 00007f0c655e41cc CR3: 0000000dd2c10000 CR4: 00000000003407e0 Apr 24 00:43:27 fir-md1-s1 kernel: Call Trace: Apr 24 00:43:27 fir-md1-s1 kernel: [] queued_spin_lock_slowpath+0xb/0xf Apr 24 00:43:27 fir-md1-s1 kernel: [] _raw_spin_lock+0x20/0x30 Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? zone_statistics+0x88/0xa0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? qsd_op_begin+0xb1/0x4b0 [lquota] Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] mdt_commitrw_write.isra.46+0x608/0xd20 [mdt] Apr 24 00:43:27 fir-md1-s1 kernel: [] mdt_obd_commitrw+0x29b/0x520 [mdt] Apr 24 00:43:27 fir-md1-s1 kernel: [] obd_commitrw+0x9c/0x370 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] tgt_brw_write+0x100d/0x1a90 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? update_curr+0x14c/0x1e0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? account_entity_dequeue+0xae/0xd0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? default_wake_function+0x12/0x20 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? __wake_up_common+0x5b/0x90 Apr 24 00:43:27 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:27 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:27 fir-md1-s1 kernel: Code: 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 14 9c 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 85 c0 <74> f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 Apr 24 00:43:27 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [mdt_io02_025:21618] Apr 24 00:43:27 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif Apr 24 00:43:27 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas Apr 24 00:43:27 fir-md1-s1 kernel: CPU: 42 PID: 21618 Comm: mdt_io02_025 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 Apr 24 00:43:27 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 Apr 24 00:43:27 fir-md1-s1 kernel: task: ffff8b732113c100 ti: ffff8b73273c8000 task.ti: ffff8b73273c8000 Apr 24 00:43:27 fir-md1-s1 kernel: RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 Apr 24 00:43:27 fir-md1-s1 kernel: RSP: 0018:ffff8b73273cb800 EFLAGS: 00000246 Apr 24 00:43:27 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fb62ca20 RCX: 0000000001510000 Apr 24 00:43:27 fir-md1-s1 kernel: RDX: ffff8b633f7db780 RSI: 0000000000f10101 RDI: ffff8b7339b45480 Apr 24 00:43:27 fir-md1-s1 kernel: RBP: ffff8b73273cb800 R08: ffff8b633f89b780 R09: 0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: R10: ffff8b633f89f140 R11: fffff658299b9c00 R12: 0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: R13: ffff8b73273cb7a0 R14: ffff8b48fb62c790 R15: 0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: FS: 00007f35da56d880(0000) GS:ffff8b633f880000(0000) knlGS:0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 24 00:43:27 fir-md1-s1 kernel: CR2: 00007f35ca9b1ff0 CR3: 0000000dd2c10000 CR4: 00000000003407e0 Apr 24 00:43:27 fir-md1-s1 kernel: Call Trace: Apr 24 00:43:27 fir-md1-s1 kernel: [] queued_spin_lock_slowpath+0xb/0xf Apr 24 00:43:27 fir-md1-s1 kernel: [] _raw_spin_lock+0x20/0x30 Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ktime_get_ts64+0x52/0xf0 Apr 24 00:43:27 fir-md1-s1 kernel: [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] mdt_obd_preprw+0x637/0x1060 [mdt] Apr 24 00:43:27 fir-md1-s1 kernel: [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? update_curr+0x14c/0x1e0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? account_entity_dequeue+0xae/0xd0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? default_wake_function+0x12/0x20 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? __wake_up_common+0x5b/0x90 Apr 24 00:43:27 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:27 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:27 fir-md1-s1 kernel: Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 14 9c 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b Apr 24 00:43:27 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#44 stuck for 22s! [mdt_io00_065:22104] Apr 24 00:43:27 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif Apr 24 00:43:27 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas Apr 24 00:43:27 fir-md1-s1 kernel: CPU: 44 PID: 22104 Comm: mdt_io00_065 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 Apr 24 00:43:27 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 Apr 24 00:43:27 fir-md1-s1 kernel: task: ffff8b733c794100 ti: ffff8b5310244000 task.ti: ffff8b5310244000 Apr 24 00:43:27 fir-md1-s1 kernel: RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x128/0x200 Apr 24 00:43:27 fir-md1-s1 kernel: RSP: 0018:ffff8b5310247800 EFLAGS: 00000246 Apr 24 00:43:27 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fb62c5f8 RCX: 0000000001610000 Apr 24 00:43:27 fir-md1-s1 kernel: RDX: ffff8b633f89b780 RSI: 0000000001510101 RDI: ffff8b7339b45480 Apr 24 00:43:27 fir-md1-s1 kernel: RBP: ffff8b5310247800 R08: ffff8b433f0db780 R09: 0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: R10: ffff8b433f0df140 R11: fffff65795efe400 R12: 0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: R13: ffff8b53102477a0 R14: ffff8b48fb62c368 R15: 0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: FS: 00007fa2ad0f3700(0000) GS:ffff8b433f0c0000(0000) knlGS:0000000000000000 Apr 24 00:43:27 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 24 00:43:27 fir-md1-s1 kernel: CR2: 00007efe23520000 CR3: 0000000dd2c10000 CR4: 00000000003407e0 Apr 24 00:43:27 fir-md1-s1 kernel: Call Trace: Apr 24 00:43:27 fir-md1-s1 kernel: [] queued_spin_lock_slowpath+0xb/0xf Apr 24 00:43:27 fir-md1-s1 kernel: [] _raw_spin_lock+0x20/0x30 Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ___slab_alloc+0x209/0x4f0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd] Apr 24 00:43:27 fir-md1-s1 kernel: [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ktime_get_ts64+0x52/0xf0 Apr 24 00:43:27 fir-md1-s1 kernel: [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] mdt_obd_preprw+0x637/0x1060 [mdt] Apr 24 00:43:27 fir-md1-s1 kernel: [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? update_curr+0x14c/0x1e0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? account_entity_dequeue+0xae/0xd0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:27 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? default_wake_function+0x12/0x20 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? __wake_up_common+0x5b/0x90 Apr 24 00:43:27 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:27 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:27 fir-md1-s1 kernel: Code: 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 14 9c 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 85 c0 <74> f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? wake_up_state+0x20/0x20 Apr 24 00:43:27 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Apr 24 00:43:27 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:27 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Apr 24 00:43:27 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:27 fir-md1-s1 kernel: Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 14 9c 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b Apr 24 00:43:31 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! [mdt_io01_006:21534] Apr 24 00:43:31 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif Apr 24 00:43:31 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas Apr 24 00:43:31 fir-md1-s1 kernel: CPU: 25 PID: 21534 Comm: mdt_io01_006 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 Apr 24 00:43:31 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 Apr 24 00:43:31 fir-md1-s1 kernel: task: ffff8b7320ff4100 ti: ffff8b7338b8c000 task.ti: ffff8b7338b8c000 Apr 24 00:43:31 fir-md1-s1 kernel: RIP: 0010:[] [] native_queued_spin_lock_slowpath+0x122/0x200 Apr 24 00:43:31 fir-md1-s1 kernel: RSP: 0018:ffff8b7338b8f800 EFLAGS: 00000246 Apr 24 00:43:31 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fce60ff0 RCX: 0000000000c90000 Apr 24 00:43:31 fir-md1-s1 kernel: RDX: ffff8b433f0db780 RSI: 0000000001610101 RDI: ffff8b7339b45480 Apr 24 00:43:31 fir-md1-s1 kernel: RBP: ffff8b7338b8f800 R08: ffff8b533f79b780 R09: 0000000000000000 Apr 24 00:43:31 fir-md1-s1 kernel: R10: ffff8b533f79f140 R11: fffff657d7e9a800 R12: 0000000000000000 Apr 24 00:43:31 fir-md1-s1 kernel: R13: ffff8b7338b8f7a0 R14: ffff8b48fce60d60 R15: 0000000000000000 Apr 24 00:43:31 fir-md1-s1 kernel: FS: 00007f43a2d5c700(0000) GS:ffff8b533f780000(0000) knlGS:0000000000000000 Apr 24 00:43:31 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 24 00:43:31 fir-md1-s1 kernel: CR2: 00007f43a2dcf000 CR3: 0000000dd2c10000 CR4: 00000000003407e0 Apr 24 00:43:31 fir-md1-s1 kernel: Call Trace: Apr 24 00:43:31 fir-md1-s1 kernel: [] queued_spin_lock_slowpath+0xb/0xf Apr 24 00:43:31 fir-md1-s1 kernel: [] _raw_spin_lock+0x20/0x30 Apr 24 00:43:31 fir-md1-s1 kernel: [] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs] Apr 24 00:43:31 fir-md1-s1 kernel: [] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs] Apr 24 00:43:31 fir-md1-s1 kernel: [] ? vsnprintf+0x234/0x6a0 Apr 24 00:43:31 fir-md1-s1 kernel: [] ldiskfs_map_blocks+0x98/0x700 [ldiskfs] Apr 24 00:43:31 fir-md1-s1 kernel: [] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs] Apr 24 00:43:31 fir-md1-s1 kernel: [] ? ktime_get_ts64+0x52/0xf0 Apr 24 00:43:31 fir-md1-s1 kernel: [] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs] Apr 24 00:43:31 fir-md1-s1 kernel: [] osd_write_prep+0x2b6/0x360 [osd_ldiskfs] Apr 24 00:43:31 fir-md1-s1 kernel: [] mdt_obd_preprw+0x637/0x1060 [mdt] Apr 24 00:43:31 fir-md1-s1 kernel: [] tgt_brw_write+0xc7e/0x1a90 [ptlrpc] Apr 24 00:43:31 fir-md1-s1 kernel: [] ? tgt_free_reply_data+0x128/0x3b0 [ptlrpc] Apr 24 00:43:31 fir-md1-s1 kernel: [] ? kfree+0x106/0x140 Apr 24 00:43:31 fir-md1-s1 kernel: [] ? tgt_free_reply_data+0x128/0x3b0 [ptlrpc] Apr 24 00:43:31 fir-md1-s1 kernel: [] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc] Apr 24 00:43:31 fir-md1-s1 kernel: [] tgt_request_handle+0xaea/0x1580 [ptlrpc] Apr 24 00:43:31 fir-md1-s1 kernel: [] ? libcfs_debug_msg+0x57/0x80 [libcfs] Apr 24 00:43:31 fir-md1-s1 kernel: [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Apr 24 00:43:31 fir-md1-s1 kernel: [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] Apr 24 00:43:31 fir-md1-s1 kernel: [] ? default_wake_function+0x12/0x20 Apr 24 00:43:31 fir-md1-s1 kernel: [] ? __wake_up_common+0x5b/0x90 Apr 24 00:43:31 fir-md1-s1 kernel: [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Apr 24 00:43:31 fir-md1-s1 kernel: [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] Apr 24 00:43:31 fir-md1-s1 kernel: [] kthread+0xd1/0xe0 Apr 24 00:43:31 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:31 fir-md1-s1 kernel: [] ret_from_fork_nospec_begin+0xe/0x21 Apr 24 00:43:31 fir-md1-s1 kernel: [] ? insert_kthread_work+0x40/0x40 Apr 24 00:43:31 fir-md1-s1 kernel: Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 14 9c 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b Apr 24 00:43:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client ac39260b-aa99-2050-abe9-06dcf51e927b (at 10.9.101.47@o2ib4) reconnecting Apr 24 00:43:31 fir-md1-s1 kernel: Lustre: Skipped 258 previous similar messages Apr 24 00:43:33 fir-md1-s1 kernel: sched: RT throttling activated Apr 24 00:43:33 fir-md1-s1 kernel: Lustre: 48105:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 16s req@ffff8b5d60ac1c50 x1631573995472736/t0(0) o4->661f0cfa-e148-dc98-69cd-517192e597e7@10.8.7.3@o2ib6:0/0 lens 4216/0 e 0 to 0 dl 0 ref 1 fl New:/2/ffffffff rc 0/-1 Apr 24 00:43:33 fir-md1-s1 kernel: Lustre: 21579:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:17s); client may timeout. req@ffff8b42da5c3050 x1630937895792336/t214891143440(0) o4->59bf854b-f25a-3380-6e0c-7f5c7b945dd1@10.9.108.15@o2ib4:15/0 lens 488/416 e 1 to 0 dl 1556091795 ref 1 fl Complete:/0/0 rc 0/0 Apr 24 00:43:33 fir-md1-s1 kernel: LustreError: 21594:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b53203bc050 x1631542737016096/t0(0) o3->be42b497-ab1b-8d58-3101-014aad577cfc@10.8.27.35@o2ib6:14/0 lens 488/440 e 1 to 0 dl 1556091794 ref 1 fl Interpret:/0/0 rc 0/0 Apr 24 00:43:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO read error with be42b497-ab1b-8d58-3101-014aad577cfc (at 10.8.27.35@o2ib6), client will retry: rc -107 Apr 24 00:43:33 fir-md1-s1 kernel: Lustre: mdt_io: This server is not able to keep up with request traffic (cpu-bound). Apr 24 00:43:33 fir-md1-s1 kernel: Lustre: 21690:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=73 reqQ=0 recA=88, svcEst=20, delay=15960 Apr 24 00:43:33 fir-md1-s1 kernel: Lustre: 21690:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8b5d5fab8850 x1631679054779200/t214891143471(0) o4->a82097ea-0a83-cc99-985b-882074216844@10.8.12.13@o2ib6:1/0 lens 3112/448 e 1 to 0 dl 1556091811 ref 2 fl Interpret:/0/0 rc 0/0 Apr 24 00:43:33 fir-md1-s1 kernel: LustreError: 21640:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.113.2@o2ib4: deadline 6:9s ago req@ffff8b5d5f463050 x1631347528787136/t0(0) o4->8829dd2a-c714-d8f5-ab1a-941339d64495@10.9.113.2@o2ib4:23/0 lens 3544/0 e 0 to 0 dl 1556091803 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Apr 24 00:43:33 fir-md1-s1 kernel: LustreError: 21610:0:(tgt_handler.c:644:process_req_last_xid()) @@@ Unexpected xid 5cbb2529bac50 vs. last_xid 5cbb2529bac5f req@ffff8b48f8e9bc50 x1631341634104400/t0(0) o4->b984ccf1-6625-41c4-2ebc-91a79a8a0ab1@10.8.23.18@o2ib6:6/0 lens 2928/0 e 0 to 0 dl 1556091816 ref 1 fl Interpret:/2/ffffffff rc 0/-1 Apr 24 00:43:33 fir-md1-s1 kernel: Lustre: 48105:0:(service.c:2011:ptlrpc_server_handle_req_in()) Skipped 24 previous similar messages Apr 24 00:45:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client cbbea467-025f-86a9-4e0e-ef1c851a3dc7 (at 10.8.26.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b42ec506000, cur 1556091909 expire 1556091759 last 1556091682 Apr 24 00:45:09 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages Apr 24 00:48:45 fir-md1-s1 kernel: Lustre: 22086:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1556092118/real 1556092118] req@ffff8b5db3396300 x1631585366495376/t0(0) o104->fir-MDT0002@10.8.12.31@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1556092125 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Apr 24 00:48:45 fir-md1-s1 kernel: Lustre: 22086:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 23 previous similar messages Apr 24 00:49:03 fir-md1-s1 kernel: Lustre: 21734:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8b620dba6c00 x1631532464055968/t0(0) o101->39fe18a4-a89c-1a84-3eb2-1fc3124ee4a0@10.9.108.28@o2ib4:8/0 lens 480/568 e 0 to 0 dl 1556092148 ref 2 fl Interpret:/0/0 rc 0/0 Apr 24 00:49:03 fir-md1-s1 kernel: Lustre: 21734:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 509 previous similar messages Apr 24 00:49:13 fir-md1-s1 kernel: LustreError: 22086:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.12.31@o2ib6) failed to reply to blocking AST (req@ffff8b5db3396300 x1631585366495376 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8b523c68b3c0/0x378007f0e0e192a1 lrc: 4/0,0 mode: PR/PR res: [0x2c001a5b3:0x636:0x0].0x0 bits 0x5b/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.8.12.31@o2ib6 remote: 0xbbfac8fcd3f5cadd expref: 3748 pid: 21505 timeout: 92010 lvb_type: 0 Apr 24 00:49:13 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.12.31@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Apr 24 00:49:13 fir-md1-s1 kernel: LustreError: 20444:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 35s: evicting client at 10.8.12.31@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8b523c68b3c0/0x378007f0e0e192a1 lrc: 3/0,0 mode: PR/PR res: [0x2c001a5b3:0x636:0x0].0x0 bits 0x5b/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.8.12.31@o2ib6 remote: 0xbbfac8fcd3f5cadd expref: 3749 pid: 21505 timeout: 0 lvb_type: 0 Apr 24 00:51:29 fir-md1-s1 kernel: Lustre: 22086:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1556092282/real 1556092282] req@ffff8b5dbb385100 x1631585368677216/t0(0) o104->fir-MDT0002@10.8.22.1@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1556092289 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Apr 24 00:51:29 fir-md1-s1 kernel: Lustre: 22086:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Apr 24 00:51:47 fir-md1-s1 kernel: Lustre: 22021:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply req@ffff8b628e6d6900 x1631535005686928/t214891595924(0) o36->6ea10cfc-48e7-f6e7-b834-4eb6674e3061@10.9.102.48@o2ib4:22/0 lens 488/3152 e 0 to 0 dl 1556092312 ref 2 fl Interpret:/0/0 rc 0/0 Apr 24 00:51:57 fir-md1-s1 kernel: LustreError: 22086:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.22.1@o2ib6) failed to reply to blocking AST (req@ffff8b5dbb385100 x1631585368677216 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8b52a68557c0/0x378007f0e0fed702 lrc: 4/0,0 mode: PR/PR res: [0x2c001a748:0x1db:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.22.1@o2ib6 remote: 0x5a75d9a873fe4860 expref: 3715 pid: 21705 timeout: 92173 lvb_type: 0 Apr 24 00:51:57 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.22.1@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Apr 24 00:51:57 fir-md1-s1 kernel: LustreError: 20444:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 35s: evicting client at 10.8.22.1@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8b52a68557c0/0x378007f0e0fed702 lrc: 3/0,0 mode: PR/PR res: [0x2c001a748:0x1db:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.22.1@o2ib6 remote: 0x5a75d9a873fe4860 expref: 3716 pid: 21705 timeout: 0 lvb_type: 0 Apr 24 00:52:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to ef4ceb90-5603-14d7-21e5-f4e7d223a0a1 (at 10.8.8.33@o2ib6) Apr 24 00:52:26 fir-md1-s1 kernel: Lustre: Skipped 287 previous similar messages Apr 24 00:53:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client f50d7e88-e046-b55a-0fe3-13cbe95d417b (at 10.8.8.33@o2ib6) reconnecting