Apr 24 00:33:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client f50d7e88-e046-b55a-0fe3-13cbe95d417b (at 10.8.8.33@o2ib6) reconnecting
Apr 24 00:33:31 fir-md1-s1 kernel: Lustre: Skipped 34 previous similar messages
Apr 24 00:42:16 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to ef4ceb90-5603-14d7-21e5-f4e7d223a0a1 (at 10.8.8.33@o2ib6)
Apr 24 00:42:16 fir-md1-s1 kernel: Lustre: Skipped 45 previous similar messages
Apr 24 00:42:58 fir-md1-s1 kernel: list passed to list_sort() too long for efficiency
Apr 24 00:43:07 fir-md1-s1 kernel: Lustre: 21207:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply
req@ffff8b37931c4800 x1631534610856048/t0(0) o101->40edf202-1dd9-d482-e1d9-d8df453c34c1@10.9.102.4@o2ib4:12/0 lens 568/0 e 1 to 0 dl 1556091792 ref 2 fl Interpret:/0/ffffffff rc 0/-1
Apr 24 00:43:07 fir-md1-s1 kernel: Lustre: 21207:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages
Apr 24 00:43:19 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#40 stuck for 23s! [mdt00_090:22029]
Apr 24 00:43:19 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif
Apr 24 00:43:19 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas
Apr 24 00:43:19 fir-md1-s1 kernel: CPU: 40 PID: 22029 Comm: mdt00_090 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1
Apr 24 00:43:19 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018
Apr 24 00:43:19 fir-md1-s1 kernel: task: ffff8b7324050000 ti: ffff8b7324ab0000 task.ti: ffff8b7324ab0000
Apr 24 00:43:19 fir-md1-s1 kernel: RIP: 0010:[<ffffffffc11f00ad>] [<ffffffffc11f00ad>] ldiskfs_inode_touch_time_cmp+0xd/0x90 [ldiskfs]
Apr 24 00:43:19 fir-md1-s1 kernel: RSP: 0018:ffff8b7324ab3310 EFLAGS: 00000282
Apr 24 00:43:19 fir-md1-s1 kernel: RAX: 8000040400080000 RBX: ffffffff9c3e3002 RCX: 00000001043969bf
Apr 24 00:43:19 fir-md1-s1 kernel: RDX: ffff8b4b1db2d278 RSI: ffff8b3d335b3138 RDI: 0000000000000000
Apr 24 00:43:19 fir-md1-s1 kernel: RBP: ffff8b7324ab3360 R08: 000000000000000a R09: 0000000000000000
Apr 24 00:43:19 fir-md1-s1 kernel: R10: 0000000000000fbe R11: ffff8b7324ab306e R12: ffff8b7324ab32f0
Apr 24 00:43:19 fir-md1-s1 kernel: R13: 0000000000000006 R14: 0000000000000032 R15: 0000000000000000
Apr 24 00:43:19 fir-md1-s1 kernel: FS: 00007f2994e3b780(0000) GS:ffff8b433f080000(0000) knlGS:0000000000000000
Apr 24 00:43:19 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 00:43:19 fir-md1-s1 kernel: CR2: 00007f2994e4c000 CR3: 0000000dd2c10000 CR4: 00000000003407e0
Apr 24 00:43:19 fir-md1-s1 kernel: Call Trace:
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b78b4f2>] ? merge+0x62/0xc0
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc11f00a0>] ? ldiskfs_init_inode_table+0x410/0x410 [ldiskfs]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b78b5eb>] list_sort+0x9b/0x250
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc11f059e>] __ldiskfs_es_shrink+0x1ce/0x2a0 [ldiskfs]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc11f0724>] ldiskfs_es_shrink+0xb4/0x130 [ldiskfs]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b5c9da5>] shrink_slab+0x175/0x340
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b5bc8af>] ? zone_watermark_ok+0x1f/0x30
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b5dde43>] ? compaction_suitable+0xa3/0xb0
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b5ccb81>] zone_reclaim+0x1d1/0x2f0
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b5c0b0b>] get_page_from_freelist+0x87b/0xa70
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b794d0d>] ? list_del+0xd/0x30
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b7824d9>] ? pointer.isra.19+0x1c9/0x4d0
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b782a14>] ? vsnprintf+0x234/0x6a0
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b5c0e76>] __alloc_pages_nodemask+0x176/0x420
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b60df68>] alloc_pages_current+0x98/0x110
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b5bb84e>] __get_free_pages+0xe/0x40
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b6195fe>] kmalloc_order_trace+0x2e/0xa0
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b61d641>] __kmalloc+0x211/0x230
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0de7f6d>] null_alloc_rs+0x16d/0x340 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0ddee06>] sptlrpc_svc_alloc_rs+0x66/0x350 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc1477771>] ? mdt_root_squash+0x21/0x430 [mdt]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0db074a>] lustre_pack_reply_v2+0x8a/0x280 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0db09af>] lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0db0b31>] lustre_pack_reply+0x11/0x20 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0dd56e3>] req_capsule_server_pack+0x43/0xf0 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc148a5cc>] mdt_getxattr+0x71c/0x12e0 [mdt]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc146bb40>] ? mdt_object_lock_internal+0x70/0x3e0 [mdt]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0dae35c>] ? lustre_msg_get_flags+0x2c/0xa0 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc1474e55>] mdt_intent_getxattr+0xc5/0x270 [mdt]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc1471a28>] mdt_intent_policy+0x2e8/0xd00 [mdt]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc1474d90>] ? mdt_intent_getattr+0x480/0x480 [mdt]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0d61d26>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc09fefa3>] ? cfs_hash_bd_add_locked+0x63/0x80 [libcfs]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0a0272e>] ? cfs_hash_add+0xbe/0x1a0 [libcfs]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0d8a6d7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0db25a0>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0e110b2>] tgt_enqueue+0x62/0x210 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0e1810a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0dbc6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0db9565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b4d67c2>] ? default_wake_function+0x12/0x20
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b4cba9b>] ? __wake_up_common+0x5b/0x90
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0dc000c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffffc0dbf510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b4c1c31>] kthread+0xd1/0xe0
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9bb74c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 24 00:43:19 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:19 fir-md1-s1 kernel: Code: ff 8d 4a 01 89 d0 f0 0f b1 0f 39 d0 0f 84 fb fd ff ff 89 c2 eb e2 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 86 e8 fc ff ff <48> 89 e5 48 c1 e8 2b a8 01 74 15 48 8b 8a e8 fc ff ff b8 01 00
Apr 24 00:43:22 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [mdt00_039:21591]
Apr 24 00:43:22 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [mdt_io01_084:22135]
Apr 24 00:43:22 fir-md1-s1 kernel: Modules linked in:
Apr 24 00:43:22 fir-md1-s1 kernel: Modules linked in:
Apr 24 00:43:22 fir-md1-s1 kernel: osp(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mdd(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lod(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mdt(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lfsck(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mgs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mgc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: osd_ldiskfs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lquota(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ldiskfs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lustre(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lmv(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mdc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: osc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lov(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: fid(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: fld(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ko2iblnd(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ptlrpc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: obdclass(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lnet(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: libcfs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: rpcsec_gss_krb5
Apr 24 00:43:22 fir-md1-s1 kernel: auth_rpcgss
Apr 24 00:43:22 fir-md1-s1 kernel: nfsv4
Apr 24 00:43:22 fir-md1-s1 kernel: dns_resolver
Apr 24 00:43:22 fir-md1-s1 kernel: nfs
Apr 24 00:43:22 fir-md1-s1 kernel: lockd
Apr 24 00:43:22 fir-md1-s1 kernel: grace
Apr 24 00:43:22 fir-md1-s1 kernel: fscache
Apr 24 00:43:22 fir-md1-s1 kernel: rdma_ucm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_ucm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: rdma_cm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: iw_cm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_ipoib(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_cm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_umad(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_fpga_tools(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_en(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_ib(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_core(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: dell_rbu
Apr 24 00:43:22 fir-md1-s1 kernel: sunrpc
Apr 24 00:43:22 fir-md1-s1 kernel: vfat
Apr 24 00:43:22 fir-md1-s1 kernel: fat
Apr 24 00:43:22 fir-md1-s1 kernel: dm_round_robin
Apr 24 00:43:22 fir-md1-s1 kernel: amd64_edac_mod
Apr 24 00:43:22 fir-md1-s1 kernel: edac_mce_amd
Apr 24 00:43:22 fir-md1-s1 kernel: kvm_amd
Apr 24 00:43:22 fir-md1-s1 kernel: kvm
Apr 24 00:43:22 fir-md1-s1 kernel: irqbypass
Apr 24 00:43:22 fir-md1-s1 kernel: crc32_pclmul
Apr 24 00:43:22 fir-md1-s1 kernel: ghash_clmulni_intel
Apr 24 00:43:22 fir-md1-s1 kernel: aesni_intel
Apr 24 00:43:22 fir-md1-s1 kernel: lrw
Apr 24 00:43:22 fir-md1-s1 kernel: gf128mul
Apr 24 00:43:22 fir-md1-s1 kernel: glue_helper
Apr 24 00:43:22 fir-md1-s1 kernel: ablk_helper
Apr 24 00:43:22 fir-md1-s1 kernel: cryptd
Apr 24 00:43:22 fir-md1-s1 kernel: dcdbas
Apr 24 00:43:22 fir-md1-s1 kernel: ses
Apr 24 00:43:22 fir-md1-s1 kernel: enclosure
Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_si
Apr 24 00:43:22 fir-md1-s1 kernel: pcspkr
Apr 24 00:43:22 fir-md1-s1 kernel: dm_multipath
Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_devintf
Apr 24 00:43:22 fir-md1-s1 kernel: dm_mod
Apr 24 00:43:22 fir-md1-s1 kernel: ccp
Apr 24 00:43:22 fir-md1-s1 kernel: sg
Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_msghandler
Apr 24 00:43:22 fir-md1-s1 kernel: k10temp
Apr 24 00:43:22 fir-md1-s1 kernel: i2c_piix4
Apr 24 00:43:22 fir-md1-s1 kernel: acpi_power_meter
Apr 24 00:43:22 fir-md1-s1 kernel: knem(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ip_tables
Apr 24 00:43:22 fir-md1-s1 kernel: ext4
Apr 24 00:43:22 fir-md1-s1 kernel: mbcache
Apr 24 00:43:22 fir-md1-s1 kernel: jbd2
Apr 24 00:43:22 fir-md1-s1 kernel: sd_mod
Apr 24 00:43:22 fir-md1-s1 kernel: crc_t10dif
Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_generic
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_ib(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_uverbs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_core(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: i2c_algo_bit
Apr 24 00:43:22 fir-md1-s1 kernel: drm_kms_helper
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_core(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: syscopyarea
Apr 24 00:43:22 fir-md1-s1 kernel: mlxfw(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: sysfillrect
Apr 24 00:43:22 fir-md1-s1 kernel: devlink
Apr 24 00:43:22 fir-md1-s1 kernel: sysimgblt
Apr 24 00:43:22 fir-md1-s1 kernel: fb_sys_fops
Apr 24 00:43:22 fir-md1-s1 kernel: mlx_compat(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ahci
Apr 24 00:43:22 fir-md1-s1 kernel: ttm
Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_pclmul
Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_common
Apr 24 00:43:22 fir-md1-s1 kernel: libahci
Apr 24 00:43:22 fir-md1-s1 kernel: drm
Apr 24 00:43:22 fir-md1-s1 kernel: tg3
Apr 24 00:43:22 fir-md1-s1 kernel: crc32c_intel
Apr 24 00:43:22 fir-md1-s1 kernel: ptp
Apr 24 00:43:22 fir-md1-s1 kernel: libata
Apr 24 00:43:22 fir-md1-s1 kernel: megaraid_sas
Apr 24 00:43:22 fir-md1-s1 kernel: drm_panel_orientation_quirks
Apr 24 00:43:22 fir-md1-s1 kernel: pps_core
Apr 24 00:43:22 fir-md1-s1 kernel: mpt3sas(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: raid_class
Apr 24 00:43:22 fir-md1-s1 kernel: scsi_transport_sas
Apr 24 00:43:22 fir-md1-s1 kernel:
Apr 24 00:43:22 fir-md1-s1 kernel: CPU: 1 PID: 22135 Comm: mdt_io01_084 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1
Apr 24 00:43:22 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018
Apr 24 00:43:22 fir-md1-s1 kernel: task: ffff8b732d24a080 ti: ffff8b732d254000 task.ti: ffff8b732d254000
Apr 24 00:43:22 fir-md1-s1 kernel: RIP: 0010:[<ffffffff9b5121e2>]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b5121e2>] native_queued_spin_lock_slowpath+0x122/0x200
Apr 24 00:43:22 fir-md1-s1 kernel: RSP: 0018:ffff8b732d257800 EFLAGS: 00000246
Apr 24 00:43:22 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b3793a33da8 RCX: 0000000000090000
Apr 24 00:43:22 fir-md1-s1 kernel: RDX: ffff8b633f69b780 RSI: 0000000000510101 RDI: ffff8b7339b45480
Apr 24 00:43:22 fir-md1-s1 kernel: RBP: ffff8b732d257800 R08: ffff8b533f61b780 R09: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: R10: ffff8b533f61f140 R11: fffff657da776800 R12: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: R13: ffff8b732d2577a0 R14: ffff8b3793a33b18 R15: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: FS: 00007f43a2cd8700(0000) GS:ffff8b533f600000(0000) knlGS:0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 00:43:22 fir-md1-s1 kernel: CR2: 00007f43a2dcf000 CR3: 0000000dd2c10000 CR4: 00000000003407e0
Apr 24 00:43:22 fir-md1-s1 kernel: Call Trace:
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb5bfcb>] queued_spin_lock_slowpath+0xb/0xf
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb6a480>] _raw_spin_lock+0x20/0x30
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11f1507>] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc1204d35>] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11c0518>] ldiskfs_map_blocks+0x98/0x700 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b501052>] ? ktime_get_ts64+0x52/0xf0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc12f6443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc12f8306>] osd_write_prep+0x2b6/0x360 [osd_ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc14a63d7>] mdt_obd_preprw+0x637/0x1060 [mdt]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e144be>] tgt_brw_write+0xc7e/0x1a90 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4e011c>] ? update_curr+0x14c/0x1e0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e0617d>] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e1810a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dbc6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0db9565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4d67c2>] ? default_wake_function+0x12/0x20
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4cba9b>] ? __wake_up_common+0x5b/0x90
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dc000c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dbf510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4c1c31>] kthread+0xd1/0xe0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb74c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:22 fir-md1-s1 kernel: Code:
Apr 24 00:43:22 fir-md1-s1 kernel: 13
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: c1
Apr 24 00:43:22 fir-md1-s1 kernel: ea
Apr 24 00:43:22 fir-md1-s1 kernel: 0d
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: 98
Apr 24 00:43:22 fir-md1-s1 kernel: 83
Apr 24 00:43:22 fir-md1-s1 kernel: e2
Apr 24 00:43:22 fir-md1-s1 kernel: 30
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: 81
Apr 24 00:43:22 fir-md1-s1 kernel: c2
Apr 24 00:43:22 fir-md1-s1 kernel: 80
Apr 24 00:43:22 fir-md1-s1 kernel: b7
Apr 24 00:43:22 fir-md1-s1 kernel: 01
Apr 24 00:43:22 fir-md1-s1 kernel: 00
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: 03
Apr 24 00:43:22 fir-md1-s1 kernel: 14
Apr 24 00:43:22 fir-md1-s1 kernel: c5
Apr 24 00:43:22 fir-md1-s1 kernel: 60
Apr 24 00:43:22 fir-md1-s1 kernel: b9
Apr 24 00:43:22 fir-md1-s1 kernel: 14
Apr 24 00:43:22 fir-md1-s1 kernel: 9c
Apr 24 00:43:22 fir-md1-s1 kernel: 4c
Apr 24 00:43:22 fir-md1-s1 kernel: 89
Apr 24 00:43:22 fir-md1-s1 kernel: 02
Apr 24 00:43:22 fir-md1-s1 kernel: 41
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel: 40
Apr 24 00:43:22 fir-md1-s1 kernel: 08
Apr 24 00:43:22 fir-md1-s1 kernel: 85
Apr 24 00:43:22 fir-md1-s1 kernel: c0
Apr 24 00:43:22 fir-md1-s1 kernel: 75
Apr 24 00:43:22 fir-md1-s1 kernel: 0f
Apr 24 00:43:22 fir-md1-s1 kernel: 0f
Apr 24 00:43:22 fir-md1-s1 kernel: 1f
Apr 24 00:43:22 fir-md1-s1 kernel: 44
Apr 24 00:43:22 fir-md1-s1 kernel: 00
Apr 24 00:43:22 fir-md1-s1 kernel: 00
Apr 24 00:43:22 fir-md1-s1 kernel: f3
Apr 24 00:43:22 fir-md1-s1 kernel: 90
Apr 24 00:43:22 fir-md1-s1 kernel: <41>
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel: 40
Apr 24 00:43:22 fir-md1-s1 kernel: 08
Apr 24 00:43:22 fir-md1-s1 kernel: 85
Apr 24 00:43:22 fir-md1-s1 kernel: c0
Apr 24 00:43:22 fir-md1-s1 kernel: 74
Apr 24 00:43:22 fir-md1-s1 kernel: f6
Apr 24 00:43:22 fir-md1-s1 kernel: 4d
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel: 08
Apr 24 00:43:22 fir-md1-s1 kernel: 4d
Apr 24 00:43:22 fir-md1-s1 kernel: 85
Apr 24 00:43:22 fir-md1-s1 kernel: c9
Apr 24 00:43:22 fir-md1-s1 kernel: 74
Apr 24 00:43:22 fir-md1-s1 kernel: 04
Apr 24 00:43:22 fir-md1-s1 kernel: 41
Apr 24 00:43:22 fir-md1-s1 kernel: 0f
Apr 24 00:43:22 fir-md1-s1 kernel: 18
Apr 24 00:43:22 fir-md1-s1 kernel: 09
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel:
Apr 24 00:43:22 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [mdt_io00_022:21608]
Apr 24 00:43:22 fir-md1-s1 kernel: Modules linked in:
Apr 24 00:43:22 fir-md1-s1 kernel: osp(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mdd(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lod(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mdt(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lfsck(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mgs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mgc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: osd_ldiskfs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lquota(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ldiskfs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lustre(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lmv(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mdc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: osc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lov(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: fid(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: fld(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ko2iblnd(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ptlrpc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: obdclass(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lnet(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: libcfs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: rpcsec_gss_krb5
Apr 24 00:43:22 fir-md1-s1 kernel: auth_rpcgss
Apr 24 00:43:22 fir-md1-s1 kernel: nfsv4
Apr 24 00:43:22 fir-md1-s1 kernel: dns_resolver
Apr 24 00:43:22 fir-md1-s1 kernel: nfs
Apr 24 00:43:22 fir-md1-s1 kernel: lockd
Apr 24 00:43:22 fir-md1-s1 kernel: grace
Apr 24 00:43:22 fir-md1-s1 kernel: fscache
Apr 24 00:43:22 fir-md1-s1 kernel: rdma_ucm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_ucm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: rdma_cm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: iw_cm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_ipoib(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_cm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_umad(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_fpga_tools(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_en(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_ib(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_core(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: dell_rbu
Apr 24 00:43:22 fir-md1-s1 kernel: sunrpc
Apr 24 00:43:22 fir-md1-s1 kernel: vfat
Apr 24 00:43:22 fir-md1-s1 kernel: fat
Apr 24 00:43:22 fir-md1-s1 kernel: dm_round_robin
Apr 24 00:43:22 fir-md1-s1 kernel: amd64_edac_mod
Apr 24 00:43:22 fir-md1-s1 kernel: edac_mce_amd
Apr 24 00:43:22 fir-md1-s1 kernel: kvm_amd
Apr 24 00:43:22 fir-md1-s1 kernel: kvm
Apr 24 00:43:22 fir-md1-s1 kernel: irqbypass
Apr 24 00:43:22 fir-md1-s1 kernel: crc32_pclmul
Apr 24 00:43:22 fir-md1-s1 kernel: ghash_clmulni_intel
Apr 24 00:43:22 fir-md1-s1 kernel: aesni_intel
Apr 24 00:43:22 fir-md1-s1 kernel: lrw
Apr 24 00:43:22 fir-md1-s1 kernel: gf128mul
Apr 24 00:43:22 fir-md1-s1 kernel: glue_helper
Apr 24 00:43:22 fir-md1-s1 kernel: ablk_helper
Apr 24 00:43:22 fir-md1-s1 kernel: cryptd
Apr 24 00:43:22 fir-md1-s1 kernel: dcdbas
Apr 24 00:43:22 fir-md1-s1 kernel: ses
Apr 24 00:43:22 fir-md1-s1 kernel: enclosure
Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_si
Apr 24 00:43:22 fir-md1-s1 kernel: pcspkr
Apr 24 00:43:22 fir-md1-s1 kernel: dm_multipath
Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_devintf
Apr 24 00:43:22 fir-md1-s1 kernel: dm_mod
Apr 24 00:43:22 fir-md1-s1 kernel: ccp
Apr 24 00:43:22 fir-md1-s1 kernel: sg
Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_msghandler
Apr 24 00:43:22 fir-md1-s1 kernel: k10temp
Apr 24 00:43:22 fir-md1-s1 kernel: i2c_piix4
Apr 24 00:43:22 fir-md1-s1 kernel: acpi_power_meter
Apr 24 00:43:22 fir-md1-s1 kernel: knem(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ip_tables
Apr 24 00:43:22 fir-md1-s1 kernel: ext4
Apr 24 00:43:22 fir-md1-s1 kernel: mbcache
Apr 24 00:43:22 fir-md1-s1 kernel: jbd2
Apr 24 00:43:22 fir-md1-s1 kernel: sd_mod
Apr 24 00:43:22 fir-md1-s1 kernel: crc_t10dif
Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_generic
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_ib(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_uverbs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_core(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: i2c_algo_bit
Apr 24 00:43:22 fir-md1-s1 kernel: drm_kms_helper
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_core(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: syscopyarea
Apr 24 00:43:22 fir-md1-s1 kernel: mlxfw(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: sysfillrect
Apr 24 00:43:22 fir-md1-s1 kernel: devlink
Apr 24 00:43:22 fir-md1-s1 kernel: sysimgblt
Apr 24 00:43:22 fir-md1-s1 kernel: fb_sys_fops
Apr 24 00:43:22 fir-md1-s1 kernel: mlx_compat(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ahci
Apr 24 00:43:22 fir-md1-s1 kernel: ttm
Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_pclmul
Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_common
Apr 24 00:43:22 fir-md1-s1 kernel: libahci
Apr 24 00:43:22 fir-md1-s1 kernel: drm
Apr 24 00:43:22 fir-md1-s1 kernel: tg3
Apr 24 00:43:22 fir-md1-s1 kernel: crc32c_intel
Apr 24 00:43:22 fir-md1-s1 kernel: ptp
Apr 24 00:43:22 fir-md1-s1 kernel: libata
Apr 24 00:43:22 fir-md1-s1 kernel: megaraid_sas
Apr 24 00:43:22 fir-md1-s1 kernel: drm_panel_orientation_quirks
Apr 24 00:43:22 fir-md1-s1 kernel: pps_core
Apr 24 00:43:22 fir-md1-s1 kernel: mpt3sas(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: raid_class
Apr 24 00:43:22 fir-md1-s1 kernel: scsi_transport_sas
Apr 24 00:43:22 fir-md1-s1 kernel:
Apr 24 00:43:22 fir-md1-s1 kernel: CPU: 4 PID: 21608 Comm: mdt_io00_022 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1
Apr 24 00:43:22 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018
Apr 24 00:43:22 fir-md1-s1 kernel: task: ffff8b7321131040 ti: ffff8b7327a18000 task.ti: ffff8b7327a18000
Apr 24 00:43:22 fir-md1-s1 kernel: RIP: 0010:[<ffffffff9b5121e2>]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b5121e2>] native_queued_spin_lock_slowpath+0x122/0x200
Apr 24 00:43:22 fir-md1-s1 kernel: RSP: 0018:ffff8b7327a1b800 EFLAGS: 00000246
Apr 24 00:43:22 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fb62b558 RCX: 0000000000210000
Apr 24 00:43:22 fir-md1-s1 kernel: RDX: ffff8b533f69b780 RSI: 0000000000490101 RDI: ffff8b7339b45480
Apr 24 00:43:22 fir-md1-s1 kernel: RBP: ffff8b7327a1b800 R08: ffff8b433ee5b780 R09: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: R10: ffff8b433ee5f140 R11: fffff657dc2bc200 R12: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: R13: ffff8b7327a1b7a0 R14: ffff8b48fb62b2c8 R15: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: FS: 00007fa2ad0f3700(0000) GS:ffff8b433ee40000(0000) knlGS:0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 00:43:22 fir-md1-s1 kernel: CR2: 00007f232944eb04 CR3: 0000000dd2c10000 CR4: 00000000003407e0
Apr 24 00:43:22 fir-md1-s1 kernel: Call Trace:
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb5bfcb>] queued_spin_lock_slowpath+0xb/0xf
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb6a480>] _raw_spin_lock+0x20/0x30
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11f1507>] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc1204d35>] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0d19632>] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11c0518>] ldiskfs_map_blocks+0x98/0x700 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b501052>] ? ktime_get_ts64+0x52/0xf0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc12f6443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc12f8306>] osd_write_prep+0x2b6/0x360 [osd_ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc14a63d7>] mdt_obd_preprw+0x637/0x1060 [mdt]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e144be>] tgt_brw_write+0xc7e/0x1a90 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4e011c>] ? update_curr+0x14c/0x1e0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e0617d>] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e1810a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dbc6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0db9565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4d67c2>] ? default_wake_function+0x12/0x20
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4cba9b>] ? __wake_up_common+0x5b/0x90
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dc000c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dbf510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4c1c31>] kthread+0xd1/0xe0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb74c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:22 fir-md1-s1 kernel: Code:
Apr 24 00:43:22 fir-md1-s1 kernel: 13
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: c1
Apr 24 00:43:22 fir-md1-s1 kernel: ea
Apr 24 00:43:22 fir-md1-s1 kernel: 0d
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: 98
Apr 24 00:43:22 fir-md1-s1 kernel: 83
Apr 24 00:43:22 fir-md1-s1 kernel: e2
Apr 24 00:43:22 fir-md1-s1 kernel: 30
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: 81
Apr 24 00:43:22 fir-md1-s1 kernel: c2
Apr 24 00:43:22 fir-md1-s1 kernel: 80
Apr 24 00:43:22 fir-md1-s1 kernel: b7
Apr 24 00:43:22 fir-md1-s1 kernel: 01
Apr 24 00:43:22 fir-md1-s1 kernel: 00
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: 03
Apr 24 00:43:22 fir-md1-s1 kernel: 14
Apr 24 00:43:22 fir-md1-s1 kernel: c5
Apr 24 00:43:22 fir-md1-s1 kernel: 60
Apr 24 00:43:22 fir-md1-s1 kernel: b9
Apr 24 00:43:22 fir-md1-s1 kernel: 14
Apr 24 00:43:22 fir-md1-s1 kernel: 9c
Apr 24 00:43:22 fir-md1-s1 kernel: 4c
Apr 24 00:43:22 fir-md1-s1 kernel: 89
Apr 24 00:43:22 fir-md1-s1 kernel: 02
Apr 24 00:43:22 fir-md1-s1 kernel: 41
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel: 40
Apr 24 00:43:22 fir-md1-s1 kernel: 08
Apr 24 00:43:22 fir-md1-s1 kernel: 85
Apr 24 00:43:22 fir-md1-s1 kernel: c0
Apr 24 00:43:22 fir-md1-s1 kernel: 75
Apr 24 00:43:22 fir-md1-s1 kernel: 0f
Apr 24 00:43:22 fir-md1-s1 kernel: 0f
Apr 24 00:43:22 fir-md1-s1 kernel: 1f
Apr 24 00:43:22 fir-md1-s1 kernel: 44
Apr 24 00:43:22 fir-md1-s1 kernel: 00
Apr 24 00:43:22 fir-md1-s1 kernel: 00
Apr 24 00:43:22 fir-md1-s1 kernel: f3
Apr 24 00:43:22 fir-md1-s1 kernel: 90
Apr 24 00:43:22 fir-md1-s1 kernel: <41>
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel: 40
Apr 24 00:43:22 fir-md1-s1 kernel: 08
Apr 24 00:43:22 fir-md1-s1 kernel: 85
Apr 24 00:43:22 fir-md1-s1 kernel: c0
Apr 24 00:43:22 fir-md1-s1 kernel: 74
Apr 24 00:43:22 fir-md1-s1 kernel: f6
Apr 24 00:43:22 fir-md1-s1 kernel: 4d
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel: 08
Apr 24 00:43:22 fir-md1-s1 kernel: 4d
Apr 24 00:43:22 fir-md1-s1 kernel: 85
Apr 24 00:43:22 fir-md1-s1 kernel: c9
Apr 24 00:43:22 fir-md1-s1 kernel: 74
Apr 24 00:43:22 fir-md1-s1 kernel: 04
Apr 24 00:43:22 fir-md1-s1 kernel: 41
Apr 24 00:43:22 fir-md1-s1 kernel: 0f
Apr 24 00:43:22 fir-md1-s1 kernel: 18
Apr 24 00:43:22 fir-md1-s1 kernel: 09
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel:
Apr 24 00:43:22 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [mdt_io01_029:21699]
Apr 24 00:43:22 fir-md1-s1 kernel: Modules linked in:
Apr 24 00:43:22 fir-md1-s1 kernel: osp(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mdd(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lod(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mdt(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lfsck(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mgs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mgc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: osd_ldiskfs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lquota(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ldiskfs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lustre(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lmv(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mdc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: osc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lov(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: fid(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: fld(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ko2iblnd(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ptlrpc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: obdclass(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lnet(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: libcfs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: rpcsec_gss_krb5
Apr 24 00:43:22 fir-md1-s1 kernel: auth_rpcgss
Apr 24 00:43:22 fir-md1-s1 kernel: nfsv4
Apr 24 00:43:22 fir-md1-s1 kernel: dns_resolver
Apr 24 00:43:22 fir-md1-s1 kernel: nfs
Apr 24 00:43:22 fir-md1-s1 kernel: lockd
Apr 24 00:43:22 fir-md1-s1 kernel: grace
Apr 24 00:43:22 fir-md1-s1 kernel: fscache
Apr 24 00:43:22 fir-md1-s1 kernel: rdma_ucm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_ucm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: rdma_cm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: iw_cm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_ipoib(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_cm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_umad(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_fpga_tools(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_en(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_ib(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_core(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: dell_rbu
Apr 24 00:43:22 fir-md1-s1 kernel: sunrpc
Apr 24 00:43:22 fir-md1-s1 kernel: vfat
Apr 24 00:43:22 fir-md1-s1 kernel: fat
Apr 24 00:43:22 fir-md1-s1 kernel: dm_round_robin
Apr 24 00:43:22 fir-md1-s1 kernel: amd64_edac_mod
Apr 24 00:43:22 fir-md1-s1 kernel: edac_mce_amd
Apr 24 00:43:22 fir-md1-s1 kernel: kvm_amd
Apr 24 00:43:22 fir-md1-s1 kernel: kvm
Apr 24 00:43:22 fir-md1-s1 kernel: irqbypass
Apr 24 00:43:22 fir-md1-s1 kernel: crc32_pclmul
Apr 24 00:43:22 fir-md1-s1 kernel: ghash_clmulni_intel
Apr 24 00:43:22 fir-md1-s1 kernel: aesni_intel
Apr 24 00:43:22 fir-md1-s1 kernel: lrw
Apr 24 00:43:22 fir-md1-s1 kernel: gf128mul
Apr 24 00:43:22 fir-md1-s1 kernel: glue_helper
Apr 24 00:43:22 fir-md1-s1 kernel: ablk_helper
Apr 24 00:43:22 fir-md1-s1 kernel: cryptd
Apr 24 00:43:22 fir-md1-s1 kernel: dcdbas
Apr 24 00:43:22 fir-md1-s1 kernel: ses
Apr 24 00:43:22 fir-md1-s1 kernel: enclosure
Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_si
Apr 24 00:43:22 fir-md1-s1 kernel: pcspkr
Apr 24 00:43:22 fir-md1-s1 kernel: dm_multipath
Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_devintf
Apr 24 00:43:22 fir-md1-s1 kernel: dm_mod
Apr 24 00:43:22 fir-md1-s1 kernel: ccp
Apr 24 00:43:22 fir-md1-s1 kernel: sg
Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_msghandler
Apr 24 00:43:22 fir-md1-s1 kernel: k10temp
Apr 24 00:43:22 fir-md1-s1 kernel: i2c_piix4
Apr 24 00:43:22 fir-md1-s1 kernel: acpi_power_meter
Apr 24 00:43:22 fir-md1-s1 kernel: knem(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ip_tables
Apr 24 00:43:22 fir-md1-s1 kernel: ext4
Apr 24 00:43:22 fir-md1-s1 kernel: mbcache
Apr 24 00:43:22 fir-md1-s1 kernel: jbd2
Apr 24 00:43:22 fir-md1-s1 kernel: sd_mod
Apr 24 00:43:22 fir-md1-s1 kernel: crc_t10dif
Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_generic
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_ib(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_uverbs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_core(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: i2c_algo_bit
Apr 24 00:43:22 fir-md1-s1 kernel: drm_kms_helper
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_core(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: syscopyarea
Apr 24 00:43:22 fir-md1-s1 kernel: mlxfw(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: sysfillrect
Apr 24 00:43:22 fir-md1-s1 kernel: devlink
Apr 24 00:43:22 fir-md1-s1 kernel: sysimgblt
Apr 24 00:43:22 fir-md1-s1 kernel: fb_sys_fops
Apr 24 00:43:22 fir-md1-s1 kernel: mlx_compat(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ahci
Apr 24 00:43:22 fir-md1-s1 kernel: ttm
Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_pclmul
Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_common
Apr 24 00:43:22 fir-md1-s1 kernel: libahci
Apr 24 00:43:22 fir-md1-s1 kernel: drm
Apr 24 00:43:22 fir-md1-s1 kernel: tg3
Apr 24 00:43:22 fir-md1-s1 kernel: crc32c_intel
Apr 24 00:43:22 fir-md1-s1 kernel: ptp
Apr 24 00:43:22 fir-md1-s1 kernel: libata
Apr 24 00:43:22 fir-md1-s1 kernel: megaraid_sas
Apr 24 00:43:22 fir-md1-s1 kernel: drm_panel_orientation_quirks
Apr 24 00:43:22 fir-md1-s1 kernel: pps_core
Apr 24 00:43:22 fir-md1-s1 kernel: mpt3sas(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: raid_class
Apr 24 00:43:22 fir-md1-s1 kernel: scsi_transport_sas
Apr 24 00:43:22 fir-md1-s1 kernel:
Apr 24 00:43:22 fir-md1-s1 kernel: CPU: 9 PID: 21699 Comm: mdt_io01_029 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1
Apr 24 00:43:22 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018
Apr 24 00:43:22 fir-md1-s1 kernel: task: ffff8b7324381040 ti: ffff8b7324388000 task.ti: ffff8b7324388000
Apr 24 00:43:22 fir-md1-s1 kernel: RIP: 0010:[<ffffffff9b5121e2>]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b5121e2>] native_queued_spin_lock_slowpath+0x122/0x200
Apr 24 00:43:22 fir-md1-s1 kernel: RSP: 0018:ffff8b732438b800 EFLAGS: 00000246
Apr 24 00:43:22 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fb670ff0 RCX: 0000000000490000
Apr 24 00:43:22 fir-md1-s1 kernel: RDX: ffff8b533f61b780 RSI: 0000000000090101 RDI: ffff8b7339b45480
Apr 24 00:43:22 fir-md1-s1 kernel: RBP: ffff8b732438b800 R08: ffff8b533f69b780 R09: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: R10: ffff8b533f69f140 R11: fffff657e292ac00 R12: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: R13: ffff8b732438b7a0 R14: ffff8b48fb670d60 R15: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: FS: 00007f43a2d5c700(0000) GS:ffff8b533f680000(0000) knlGS:0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 00:43:22 fir-md1-s1 kernel: CR2: 00007f43a2dcf000 CR3: 0000000dd2c10000 CR4: 00000000003407e0
Apr 24 00:43:22 fir-md1-s1 kernel: Call Trace:
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb5bfcb>] queued_spin_lock_slowpath+0xb/0xf
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb6a480>] _raw_spin_lock+0x20/0x30
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11f1507>] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc1204d35>] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0d19632>] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11c0518>] ldiskfs_map_blocks+0x98/0x700 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b501052>] ? ktime_get_ts64+0x52/0xf0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc12f6443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc12f8306>] osd_write_prep+0x2b6/0x360 [osd_ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc14a63d7>] mdt_obd_preprw+0x637/0x1060 [mdt]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e144be>] tgt_brw_write+0xc7e/0x1a90 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4e011c>] ? update_curr+0x14c/0x1e0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e0617d>] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e1810a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dbc6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0db9565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4d67c2>] ? default_wake_function+0x12/0x20
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4cba9b>] ? __wake_up_common+0x5b/0x90
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dc000c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dbf510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4c1c31>] kthread+0xd1/0xe0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb74c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:22 fir-md1-s1 kernel: Code:
Apr 24 00:43:22 fir-md1-s1 kernel: 13
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: c1
Apr 24 00:43:22 fir-md1-s1 kernel: ea
Apr 24 00:43:22 fir-md1-s1 kernel: 0d
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: 98
Apr 24 00:43:22 fir-md1-s1 kernel: 83
Apr 24 00:43:22 fir-md1-s1 kernel: e2
Apr 24 00:43:22 fir-md1-s1 kernel: 30
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: 81
Apr 24 00:43:22 fir-md1-s1 kernel: c2
Apr 24 00:43:22 fir-md1-s1 kernel: 80
Apr 24 00:43:22 fir-md1-s1 kernel: b7
Apr 24 00:43:22 fir-md1-s1 kernel: 01
Apr 24 00:43:22 fir-md1-s1 kernel: 00
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: 03
Apr 24 00:43:22 fir-md1-s1 kernel: 14
Apr 24 00:43:22 fir-md1-s1 kernel: c5
Apr 24 00:43:22 fir-md1-s1 kernel: 60
Apr 24 00:43:22 fir-md1-s1 kernel: b9
Apr 24 00:43:22 fir-md1-s1 kernel: 14
Apr 24 00:43:22 fir-md1-s1 kernel: 9c
Apr 24 00:43:22 fir-md1-s1 kernel: 4c
Apr 24 00:43:22 fir-md1-s1 kernel: 89
Apr 24 00:43:22 fir-md1-s1 kernel: 02
Apr 24 00:43:22 fir-md1-s1 kernel: 41
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel: 40
Apr 24 00:43:22 fir-md1-s1 kernel: 08
Apr 24 00:43:22 fir-md1-s1 kernel: 85
Apr 24 00:43:22 fir-md1-s1 kernel: c0
Apr 24 00:43:22 fir-md1-s1 kernel: 75
Apr 24 00:43:22 fir-md1-s1 kernel: 0f
Apr 24 00:43:22 fir-md1-s1 kernel: 0f
Apr 24 00:43:22 fir-md1-s1 kernel: 1f
Apr 24 00:43:22 fir-md1-s1 kernel: 44
Apr 24 00:43:22 fir-md1-s1 kernel: 00
Apr 24 00:43:22 fir-md1-s1 kernel: 00
Apr 24 00:43:22 fir-md1-s1 kernel: f3
Apr 24 00:43:22 fir-md1-s1 kernel: 90
Apr 24 00:43:22 fir-md1-s1 kernel: <41>
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel: 40
Apr 24 00:43:22 fir-md1-s1 kernel: 08
Apr 24 00:43:22 fir-md1-s1 kernel: 85
Apr 24 00:43:22 fir-md1-s1 kernel: c0
Apr 24 00:43:22 fir-md1-s1 kernel: 74
Apr 24 00:43:22 fir-md1-s1 kernel: f6
Apr 24 00:43:22 fir-md1-s1 kernel: 4d
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel: 08
Apr 24 00:43:22 fir-md1-s1 kernel: 4d
Apr 24 00:43:22 fir-md1-s1 kernel: 85
Apr 24 00:43:22 fir-md1-s1 kernel: c9
Apr 24 00:43:22 fir-md1-s1 kernel: 74
Apr 24 00:43:22 fir-md1-s1 kernel: 04
Apr 24 00:43:22 fir-md1-s1 kernel: 41
Apr 24 00:43:22 fir-md1-s1 kernel: 0f
Apr 24 00:43:22 fir-md1-s1 kernel: 18
Apr 24 00:43:22 fir-md1-s1 kernel: 09
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel:
Apr 24 00:43:22 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [mdt_io02_030:21634]
Apr 24 00:43:22 fir-md1-s1 kernel: Modules linked in:
Apr 24 00:43:22 fir-md1-s1 kernel: osp(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mdd(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lod(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mdt(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lfsck(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mgs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mgc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: osd_ldiskfs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lquota(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ldiskfs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lustre(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lmv(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mdc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: osc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lov(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: fid(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: fld(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ko2iblnd(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ptlrpc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: obdclass(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lnet(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: libcfs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: rpcsec_gss_krb5
Apr 24 00:43:22 fir-md1-s1 kernel: auth_rpcgss
Apr 24 00:43:22 fir-md1-s1 kernel: nfsv4
Apr 24 00:43:22 fir-md1-s1 kernel: dns_resolver
Apr 24 00:43:22 fir-md1-s1 kernel: nfs
Apr 24 00:43:22 fir-md1-s1 kernel: lockd
Apr 24 00:43:22 fir-md1-s1 kernel: grace
Apr 24 00:43:22 fir-md1-s1 kernel: fscache
Apr 24 00:43:22 fir-md1-s1 kernel: rdma_ucm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_ucm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: rdma_cm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: iw_cm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_ipoib(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_cm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_umad(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_fpga_tools(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_en(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_ib(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_core(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: dell_rbu
Apr 24 00:43:22 fir-md1-s1 kernel: sunrpc
Apr 24 00:43:22 fir-md1-s1 kernel: vfat
Apr 24 00:43:22 fir-md1-s1 kernel: fat
Apr 24 00:43:22 fir-md1-s1 kernel: dm_round_robin
Apr 24 00:43:22 fir-md1-s1 kernel: amd64_edac_mod
Apr 24 00:43:22 fir-md1-s1 kernel: edac_mce_amd
Apr 24 00:43:22 fir-md1-s1 kernel: kvm_amd
Apr 24 00:43:22 fir-md1-s1 kernel: kvm
Apr 24 00:43:22 fir-md1-s1 kernel: irqbypass
Apr 24 00:43:22 fir-md1-s1 kernel: crc32_pclmul
Apr 24 00:43:22 fir-md1-s1 kernel: ghash_clmulni_intel
Apr 24 00:43:22 fir-md1-s1 kernel: aesni_intel
Apr 24 00:43:22 fir-md1-s1 kernel: lrw
Apr 24 00:43:22 fir-md1-s1 kernel: gf128mul
Apr 24 00:43:22 fir-md1-s1 kernel: glue_helper
Apr 24 00:43:22 fir-md1-s1 kernel: ablk_helper
Apr 24 00:43:22 fir-md1-s1 kernel: cryptd
Apr 24 00:43:22 fir-md1-s1 kernel: dcdbas
Apr 24 00:43:22 fir-md1-s1 kernel: ses
Apr 24 00:43:22 fir-md1-s1 kernel: enclosure
Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_si
Apr 24 00:43:22 fir-md1-s1 kernel: pcspkr
Apr 24 00:43:22 fir-md1-s1 kernel: dm_multipath
Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_devintf
Apr 24 00:43:22 fir-md1-s1 kernel: dm_mod
Apr 24 00:43:22 fir-md1-s1 kernel: ccp
Apr 24 00:43:22 fir-md1-s1 kernel: sg
Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_msghandler
Apr 24 00:43:22 fir-md1-s1 kernel: k10temp
Apr 24 00:43:22 fir-md1-s1 kernel: i2c_piix4
Apr 24 00:43:22 fir-md1-s1 kernel: acpi_power_meter
Apr 24 00:43:22 fir-md1-s1 kernel: knem(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ip_tables
Apr 24 00:43:22 fir-md1-s1 kernel: ext4
Apr 24 00:43:22 fir-md1-s1 kernel: mbcache
Apr 24 00:43:22 fir-md1-s1 kernel: jbd2
Apr 24 00:43:22 fir-md1-s1 kernel: sd_mod
Apr 24 00:43:22 fir-md1-s1 kernel: crc_t10dif
Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_generic
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_ib(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_uverbs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_core(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: i2c_algo_bit
Apr 24 00:43:22 fir-md1-s1 kernel: drm_kms_helper
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_core(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: syscopyarea
Apr 24 00:43:22 fir-md1-s1 kernel: mlxfw(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: sysfillrect
Apr 24 00:43:22 fir-md1-s1 kernel: devlink
Apr 24 00:43:22 fir-md1-s1 kernel: sysimgblt
Apr 24 00:43:22 fir-md1-s1 kernel: fb_sys_fops
Apr 24 00:43:22 fir-md1-s1 kernel: mlx_compat(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ahci
Apr 24 00:43:22 fir-md1-s1 kernel: ttm
Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_pclmul
Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_common
Apr 24 00:43:22 fir-md1-s1 kernel: libahci
Apr 24 00:43:22 fir-md1-s1 kernel: drm
Apr 24 00:43:22 fir-md1-s1 kernel: tg3
Apr 24 00:43:22 fir-md1-s1 kernel: crc32c_intel
Apr 24 00:43:22 fir-md1-s1 kernel: ptp
Apr 24 00:43:22 fir-md1-s1 kernel: libata
Apr 24 00:43:22 fir-md1-s1 kernel: megaraid_sas
Apr 24 00:43:22 fir-md1-s1 kernel: drm_panel_orientation_quirks
Apr 24 00:43:22 fir-md1-s1 kernel: pps_core
Apr 24 00:43:22 fir-md1-s1 kernel: mpt3sas(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: raid_class
Apr 24 00:43:22 fir-md1-s1 kernel: scsi_transport_sas
Apr 24 00:43:22 fir-md1-s1 kernel:
Apr 24 00:43:22 fir-md1-s1 kernel: CPU: 10 PID: 21634 Comm: mdt_io02_030 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1
Apr 24 00:43:22 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018
Apr 24 00:43:22 fir-md1-s1 kernel: task: ffff8b7328f86180 ti: ffff8b732256c000 task.ti: ffff8b732256c000
Apr 24 00:43:22 fir-md1-s1 kernel: RIP: 0010:[<ffffffff9b5121e2>]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b5121e2>] native_queued_spin_lock_slowpath+0x122/0x200
Apr 24 00:43:22 fir-md1-s1 kernel: RSP: 0018:ffff8b732256f800 EFLAGS: 00000246
Apr 24 00:43:22 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b5d5fe93da8 RCX: 0000000000510000
Apr 24 00:43:22 fir-md1-s1 kernel: RDX: ffff8b433f05b780 RSI: 0000000001210101 RDI: ffff8b7339b45480
Apr 24 00:43:22 fir-md1-s1 kernel: RBP: ffff8b732256f800 R08: ffff8b633f69b780 R09: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: R10: ffff8b633f69f140 R11: fffff6582b75a000 R12: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: R13: ffff8b732256f7a0 R14: ffff8b5d5fe93b18 R15: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: FS: 00007fadc53ed700(0000) GS:ffff8b633f680000(0000) knlGS:0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 00:43:22 fir-md1-s1 kernel: CR2: 00007fadc7f4d000 CR3: 0000000dd2c10000 CR4: 00000000003407e0
Apr 24 00:43:22 fir-md1-s1 kernel: Call Trace:
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb5bfcb>] queued_spin_lock_slowpath+0xb/0xf
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb6a480>] _raw_spin_lock+0x20/0x30
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11f1507>] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc1204d35>] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b782a14>] ? vsnprintf+0x234/0x6a0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11c0518>] ldiskfs_map_blocks+0x98/0x700 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b501052>] ? ktime_get_ts64+0x52/0xf0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc12f6443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc12f8306>] osd_write_prep+0x2b6/0x360 [osd_ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc14a63d7>] mdt_obd_preprw+0x637/0x1060 [mdt]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e144be>] tgt_brw_write+0xc7e/0x1a90 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4e011c>] ? update_curr+0x14c/0x1e0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e0617d>] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e1810a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dbc6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0db9565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4d67c2>] ? default_wake_function+0x12/0x20
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4cba9b>] ? __wake_up_common+0x5b/0x90
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dc000c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dbf510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4c1c31>] kthread+0xd1/0xe0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb74c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:22 fir-md1-s1 kernel: Code:
Apr 24 00:43:22 fir-md1-s1 kernel: 13
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: c1
Apr 24 00:43:22 fir-md1-s1 kernel: ea
Apr 24 00:43:22 fir-md1-s1 kernel: 0d
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: 98
Apr 24 00:43:22 fir-md1-s1 kernel: 83
Apr 24 00:43:22 fir-md1-s1 kernel: e2
Apr 24 00:43:22 fir-md1-s1 kernel: 30
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: 81
Apr 24 00:43:22 fir-md1-s1 kernel: c2
Apr 24 00:43:22 fir-md1-s1 kernel: 80
Apr 24 00:43:22 fir-md1-s1 kernel: b7
Apr 24 00:43:22 fir-md1-s1 kernel: 01
Apr 24 00:43:22 fir-md1-s1 kernel: 00
Apr 24 00:43:22 fir-md1-s1 kernel: 48
Apr 24 00:43:22 fir-md1-s1 kernel: 03
Apr 24 00:43:22 fir-md1-s1 kernel: 14
Apr 24 00:43:22 fir-md1-s1 kernel: c5
Apr 24 00:43:22 fir-md1-s1 kernel: 60
Apr 24 00:43:22 fir-md1-s1 kernel: b9
Apr 24 00:43:22 fir-md1-s1 kernel: 14
Apr 24 00:43:22 fir-md1-s1 kernel: 9c
Apr 24 00:43:22 fir-md1-s1 kernel: 4c
Apr 24 00:43:22 fir-md1-s1 kernel: 89
Apr 24 00:43:22 fir-md1-s1 kernel: 02
Apr 24 00:43:22 fir-md1-s1 kernel: 41
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel: 40
Apr 24 00:43:22 fir-md1-s1 kernel: 08
Apr 24 00:43:22 fir-md1-s1 kernel: 85
Apr 24 00:43:22 fir-md1-s1 kernel: c0
Apr 24 00:43:22 fir-md1-s1 kernel: 75
Apr 24 00:43:22 fir-md1-s1 kernel: 0f
Apr 24 00:43:22 fir-md1-s1 kernel: 0f
Apr 24 00:43:22 fir-md1-s1 kernel: 1f
Apr 24 00:43:22 fir-md1-s1 kernel: 44
Apr 24 00:43:22 fir-md1-s1 kernel: 00
Apr 24 00:43:22 fir-md1-s1 kernel: 00
Apr 24 00:43:22 fir-md1-s1 kernel: f3
Apr 24 00:43:22 fir-md1-s1 kernel: 90
Apr 24 00:43:22 fir-md1-s1 kernel: <41>
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel: 40
Apr 24 00:43:22 fir-md1-s1 kernel: 08
Apr 24 00:43:22 fir-md1-s1 kernel: 85
Apr 24 00:43:22 fir-md1-s1 kernel: c0
Apr 24 00:43:22 fir-md1-s1 kernel: 74
Apr 24 00:43:22 fir-md1-s1 kernel: f6
Apr 24 00:43:22 fir-md1-s1 kernel: 4d
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel: 08
Apr 24 00:43:22 fir-md1-s1 kernel: 4d
Apr 24 00:43:22 fir-md1-s1 kernel: 85
Apr 24 00:43:22 fir-md1-s1 kernel: c9
Apr 24 00:43:22 fir-md1-s1 kernel: 74
Apr 24 00:43:22 fir-md1-s1 kernel: 04
Apr 24 00:43:22 fir-md1-s1 kernel: 41
Apr 24 00:43:22 fir-md1-s1 kernel: 0f
Apr 24 00:43:22 fir-md1-s1 kernel: 18
Apr 24 00:43:22 fir-md1-s1 kernel: 09
Apr 24 00:43:22 fir-md1-s1 kernel: 8b
Apr 24 00:43:22 fir-md1-s1 kernel:
Apr 24 00:43:22 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [mdt_io01_001:20782]
Apr 24 00:43:22 fir-md1-s1 kernel: Modules linked in:
Apr 24 00:43:22 fir-md1-s1 kernel: osp(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mdd(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lod(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mdt(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lfsck(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mgs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mgc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: osd_ldiskfs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lquota(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ldiskfs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lustre(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lmv(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mdc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: osc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lov(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: fid(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: fld(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ko2iblnd(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ptlrpc(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: obdclass(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: lnet(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: libcfs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: rpcsec_gss_krb5
Apr 24 00:43:22 fir-md1-s1 kernel: auth_rpcgss
Apr 24 00:43:22 fir-md1-s1 kernel: nfsv4
Apr 24 00:43:22 fir-md1-s1 kernel: dns_resolver
Apr 24 00:43:22 fir-md1-s1 kernel: nfs
Apr 24 00:43:22 fir-md1-s1 kernel: lockd
Apr 24 00:43:22 fir-md1-s1 kernel: grace
Apr 24 00:43:22 fir-md1-s1 kernel: fscache
Apr 24 00:43:22 fir-md1-s1 kernel: rdma_ucm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_ucm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: rdma_cm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: iw_cm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_ipoib(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_cm(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_umad(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_fpga_tools(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_en(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_ib(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: mlx4_core(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: dell_rbu
Apr 24 00:43:22 fir-md1-s1 kernel: sunrpc
Apr 24 00:43:22 fir-md1-s1 kernel: vfat
Apr 24 00:43:22 fir-md1-s1 kernel: fat
Apr 24 00:43:22 fir-md1-s1 kernel: dm_round_robin
Apr 24 00:43:22 fir-md1-s1 kernel: amd64_edac_mod
Apr 24 00:43:22 fir-md1-s1 kernel: edac_mce_amd
Apr 24 00:43:22 fir-md1-s1 kernel: kvm_amd
Apr 24 00:43:22 fir-md1-s1 kernel: kvm
Apr 24 00:43:22 fir-md1-s1 kernel: irqbypass
Apr 24 00:43:22 fir-md1-s1 kernel: crc32_pclmul
Apr 24 00:43:22 fir-md1-s1 kernel: ghash_clmulni_intel
Apr 24 00:43:22 fir-md1-s1 kernel: aesni_intel
Apr 24 00:43:22 fir-md1-s1 kernel: lrw
Apr 24 00:43:22 fir-md1-s1 kernel: gf128mul
Apr 24 00:43:22 fir-md1-s1 kernel: glue_helper
Apr 24 00:43:22 fir-md1-s1 kernel: ablk_helper
Apr 24 00:43:22 fir-md1-s1 kernel: cryptd
Apr 24 00:43:22 fir-md1-s1 kernel: dcdbas
Apr 24 00:43:22 fir-md1-s1 kernel: ses
Apr 24 00:43:22 fir-md1-s1 kernel: enclosure
Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_si
Apr 24 00:43:22 fir-md1-s1 kernel: pcspkr
Apr 24 00:43:22 fir-md1-s1 kernel: dm_multipath
Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_devintf
Apr 24 00:43:22 fir-md1-s1 kernel: dm_mod
Apr 24 00:43:22 fir-md1-s1 kernel: ccp
Apr 24 00:43:22 fir-md1-s1 kernel: sg
Apr 24 00:43:22 fir-md1-s1 kernel: ipmi_msghandler
Apr 24 00:43:22 fir-md1-s1 kernel: k10temp
Apr 24 00:43:22 fir-md1-s1 kernel: i2c_piix4
Apr 24 00:43:22 fir-md1-s1 kernel: acpi_power_meter
Apr 24 00:43:22 fir-md1-s1 kernel: knem(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ip_tables
Apr 24 00:43:22 fir-md1-s1 kernel: ext4
Apr 24 00:43:22 fir-md1-s1 kernel: mbcache
Apr 24 00:43:22 fir-md1-s1 kernel: jbd2
Apr 24 00:43:22 fir-md1-s1 kernel: sd_mod
Apr 24 00:43:22 fir-md1-s1 kernel: crc_t10dif
Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_generic
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_ib(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_uverbs(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ib_core(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: i2c_algo_bit
Apr 24 00:43:22 fir-md1-s1 kernel: drm_kms_helper
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_core(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: syscopyarea
Apr 24 00:43:22 fir-md1-s1 kernel: mlxfw(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: sysfillrect
Apr 24 00:43:22 fir-md1-s1 kernel: devlink
Apr 24 00:43:22 fir-md1-s1 kernel: sysimgblt
Apr 24 00:43:22 fir-md1-s1 kernel: fb_sys_fops
Apr 24 00:43:22 fir-md1-s1 kernel: mlx_compat(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: ahci
Apr 24 00:43:22 fir-md1-s1 kernel: ttm
Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_pclmul
Apr 24 00:43:22 fir-md1-s1 kernel: crct10dif_common
Apr 24 00:43:22 fir-md1-s1 kernel: libahci
Apr 24 00:43:22 fir-md1-s1 kernel: drm
Apr 24 00:43:22 fir-md1-s1 kernel: tg3
Apr 24 00:43:22 fir-md1-s1 kernel: crc32c_intel
Apr 24 00:43:22 fir-md1-s1 kernel: ptp
Apr 24 00:43:22 fir-md1-s1 kernel: libata
Apr 24 00:43:22 fir-md1-s1 kernel: megaraid_sas
Apr 24 00:43:22 fir-md1-s1 kernel: drm_panel_orientation_quirks
Apr 24 00:43:22 fir-md1-s1 kernel: pps_core
Apr 24 00:43:22 fir-md1-s1 kernel: mpt3sas(OE)
Apr 24 00:43:22 fir-md1-s1 kernel: raid_class
Apr 24 00:43:22 fir-md1-s1 kernel: scsi_transport_sas
Apr 24 00:43:22 fir-md1-s1 kernel:
Apr 24 00:43:22 fir-md1-s1 kernel: CPU: 13 PID: 20782 Comm: mdt_io01_001 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1
Apr 24 00:43:22 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018
Apr 24 00:43:22 fir-md1-s1 kernel: task: ffff8b7330309040 ti: ffff8b73392ac000 task.ti: ffff8b73392ac000
Apr 24 00:43:22 fir-md1-s1 kernel: RIP: 0010:[<ffffffff9b5121e2>]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b5121e2>] native_queued_spin_lock_slowpath+0x122/0x200
Apr 24 00:43:22 fir-md1-s1 kernel: RSP: 0018:ffff8b73392af800 EFLAGS: 00000246
Apr 24 00:43:22 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fb62c1d0 RCX: 0000000000690000
Apr 24 00:43:22 fir-md1-s1 kernel: RDX: ffff8b433ee5b780 RSI: 0000000000210101 RDI: ffff8b7339b45480
Apr 24 00:43:22 fir-md1-s1 kernel: RBP: ffff8b73392af800 R08: ffff8b533f6db780 R09: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: R10: ffff8b533f6df140 R11: fffff657da9eda00 R12: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: R13: ffff8b73392af7a0 R14: ffff8b48fb62bf40 R15: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: FS: 00007f1966f48880(0000) GS:ffff8b533f6c0000(0000) knlGS:0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 00:43:22 fir-md1-s1 kernel: CR2: 00007f1530508764 CR3: 0000002031f6e000 CR4: 00000000003407e0
Apr 24 00:43:22 fir-md1-s1 kernel: Call Trace:
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb5bfcb>] queued_spin_lock_slowpath+0xb/0xf
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb6a480>] _raw_spin_lock+0x20/0x30
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11f1507>] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc1204d35>] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0d19632>] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11c0518>] ldiskfs_map_blocks+0x98/0x700 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b501052>] ? ktime_get_ts64+0x52/0xf0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc12f6443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc12f8306>] osd_write_prep+0x2b6/0x360 [osd_ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc14a63d7>] mdt_obd_preprw+0x637/0x1060 [mdt]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e144be>] tgt_brw_write+0xc7e/0x1a90 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4e011c>] ? update_curr+0x14c/0x1e0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e0617d>] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e1810a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dbc6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0db9565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic
Apr 24 00:43:22 fir-md1-s1 kernel: mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas
Apr 24 00:43:22 fir-md1-s1 kernel: CPU: 0 PID: 21591 Comm: mdt00_039 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1
Apr 24 00:43:22 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018
Apr 24 00:43:22 fir-md1-s1 kernel: task: ffff8b7322485140 ti: ffff8b7324208000 task.ti: ffff8b7324208000
Apr 24 00:43:22 fir-md1-s1 kernel: RIP: 0010:[<ffffffff9b512216>] [<ffffffff9b512216>] native_queued_spin_lock_slowpath+0x156/0x200
Apr 24 00:43:22 fir-md1-s1 kernel: RSP: 0018:ffff8b732420b508 EFLAGS: 00000202
Apr 24 00:43:22 fir-md1-s1 kernel: RAX: 0000000000000101 RBX: ffff8b3793b82db8 RCX: 0000000000010000
Apr 24 00:43:22 fir-md1-s1 kernel: RDX: 0000000000410101 RSI: 0000000000000101 RDI: ffff8b7339b45480
Apr 24 00:43:22 fir-md1-s1 kernel: RBP: ffff8b732420b508 R08: ffff8b433ee1b780 R09: 0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: R10: ffff8b433ee1f140 R11: fffff6579bdefa00 R12: ffff8b732420b4a8
Apr 24 00:43:22 fir-md1-s1 kernel: R13: ffff8b3793b82ea0 R14: 0000000000000000 R15: 1fffffffffffffff
Apr 24 00:43:22 fir-md1-s1 kernel: FS: 00007f73d2277880(0000) GS:ffff8b433ee00000(0000) knlGS:0000000000000000
Apr 24 00:43:22 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 00:43:22 fir-md1-s1 kernel: CR2: 0000000000485110 CR3: 0000001038288000 CR4: 00000000003407f0
Apr 24 00:43:22 fir-md1-s1 kernel: Call Trace:
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb5bfcb>] queued_spin_lock_slowpath+0xb/0xf
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb6a480>] _raw_spin_lock+0x20/0x30
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11f1507>] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc1204d35>] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0b56a51>] ? lprocfs_counter_sub+0xc1/0x130 [obdclass]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc130b5d4>] ? dynlock_unlock+0x194/0x1e0 [osd_ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b67882d>] ? __brelse+0x3d/0x50
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc12efd92>] ? iam_path_release+0x42/0x60 [osd_ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11c0518>] ldiskfs_map_blocks+0x98/0x700 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11c0be5>] ldiskfs_getblk+0x65/0x200 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11c0da7>] ldiskfs_bread+0x27/0xc0 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11b4c41>] ldiskfs_append+0x81/0x150 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11bcd8f>] ldiskfs_init_new_dir+0xcf/0x230 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b67882d>] ? __brelse+0x3d/0x50
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc11bcf3e>] ldiskfs_add_dot_dotdot+0x4e/0x90 [ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc12cd6af>] osd_add_dot_dotdot_internal.isra.77+0x5f/0x80 [osd_ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc12e3a7a>] osd_index_ea_insert+0xb1a/0x1240 [osd_ldiskfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc159a271>] lod_sub_insert+0x1c1/0x340 [lod]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc1575a54>] lod_insert+0x24/0x30 [lod]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc15e9bcc>] __mdd_index_insert_only+0x1cc/0x280 [mdd]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc15ebd78>] mdd_create_object+0x6c8/0x820 [mdd]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc15f5c40>] mdd_create+0xd80/0x1440 [mdd]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc1485b94>] mdt_create+0xb54/0x1090 [mdt]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0b8e321>] ? lprocfs_job_stats_log+0xd1/0x640 [obdclass]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc148623b>] mdt_reint_create+0x16b/0x360 [mdt]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc1488ed3>] mdt_reint_rec+0x83/0x210 [mdt]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc1467143>] mdt_reint_internal+0x6e3/0xaf0 [mdt]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc146f404>] ? mdt_thread_info_init+0xa4/0x1e0 [mdt]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc14724a7>] mdt_reint+0x67/0x140 [mdt]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0e1810a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dbc6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0db9565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4d67c2>] ? default_wake_function+0x12/0x20
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4cba9b>] ? __wake_up_common+0x5b/0x90
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dc000c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffffc0dbf510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4c1c31>] kthread+0xd1/0xe0
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9bb74c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 24 00:43:22 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:22 fir-md1-s1 kernel: Code: 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 <8b> 17 0f b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84
Apr 24 00:43:27 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [mdt_io01_016:21594]
Apr 24 00:43:27 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif
Apr 24 00:43:27 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas
Apr 24 00:43:27 fir-md1-s1 kernel: CPU: 17 PID: 21594 Comm: mdt_io01_016 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1
Apr 24 00:43:27 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018
Apr 24 00:43:27 fir-md1-s1 kernel: task: ffff8b733ecd1040 ti: ffff8b733ecdc000 task.ti: ffff8b733ecdc000
Apr 24 00:43:27 fir-md1-s1 kernel: RIP: 0010:[<ffffffff9b51228e>] [<ffffffff9b51228e>] native_queued_spin_lock_slowpath+0x1ce/0x200
Apr 24 00:43:27 fir-md1-s1 kernel: RSP: 0018:ffff8b733ecdf790 EFLAGS: 00000202
Apr 24 00:43:27 fir-md1-s1 kernel: RAX: 0000000000000001 RBX: ffff8b733ecdf780 RCX: 0000000000000001
Apr 24 00:43:27 fir-md1-s1 kernel: RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff8b7339b45480
Apr 24 00:43:27 fir-md1-s1 kernel: RBP: ffff8b733ecdf790 R08: 0000000000000101 R09: ffffffffc1204d1a
Apr 24 00:43:27 fir-md1-s1 kernel: R10: ffff8b533f71f140 R11: fffff657dd4db400 R12: 0000005300000000
Apr 24 00:43:27 fir-md1-s1 kernel: R13: ffff8b733ecdf70e R14: 0000000a0000ffff R15: ffffffff9b781153
Apr 24 00:43:27 fir-md1-s1 kernel: FS: 00007f43a2d5c700(0000) GS:ffff8b533f700000(0000) knlGS:0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 00:43:27 fir-md1-s1 kernel: CR2: 00007f43a2dcf000 CR3: 0000000dd2c10000 CR4: 00000000003407e0
Apr 24 00:43:27 fir-md1-s1 kernel: Call Trace:
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb5bfcb>] queued_spin_lock_slowpath+0xb/0xf
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb6a480>] _raw_spin_lock+0x20/0x30
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc11f1507>] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc1204d35>] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0350b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc11c0518>] ldiskfs_map_blocks+0x98/0x700 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0d19632>] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc12f6443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc12f766e>] osd_read_prep+0x2de/0x400 [osd_ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc14a6aff>] mdt_obd_preprw+0xd5f/0x1060 [mdt]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b780152>] ? put_dec+0x72/0x90
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e1957b>] tgt_brw_read+0x9db/0x1e50 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b782a14>] ? vsnprintf+0x234/0x6a0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0b56929>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0de7f86>] ? null_alloc_rs+0x186/0x340 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0db080f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0db09af>] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0db0b31>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e1810a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dbc6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0db9565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4d67c2>] ? default_wake_function+0x12/0x20
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4cba9b>] ? __wake_up_common+0x5b/0x90
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dc000c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dbf510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1c31>] kthread+0xd1/0xe0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb74c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:27 fir-md1-s1 kernel: Code: 37 81 fe 00 01 00 00 74 f4 e9 93 fe ff ff 0f 1f 80 00 00 00 00 83 fa 01 75 11 0f 1f 00 e9 68 fe ff ff 0f 1f 00 85 c0 74 0c f3 90 <8b> 07 0f b6 c0 83 f8 03 75 f0 b8 01 00 00 00 66 89 07 5d c3 66
Apr 24 00:43:27 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [mdt_io02_028:21630]
Apr 24 00:43:27 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif
Apr 24 00:43:27 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas
Apr 24 00:43:27 fir-md1-s1 kernel: CPU: 30 PID: 21630 Comm: mdt_io02_028 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1
Apr 24 00:43:27 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018
Apr 24 00:43:27 fir-md1-s1 kernel: task: ffff8b7328f82080 ti: ffff8b7321174000 task.ti: ffff8b7321174000
Apr 24 00:43:27 fir-md1-s1 kernel: RIP: 0010:[<ffffffff9b5121e2>] [<ffffffff9b5121e2>] native_queued_spin_lock_slowpath+0x122/0x200
Apr 24 00:43:27 fir-md1-s1 kernel: RSP: 0018:ffff8b7321177800 EFLAGS: 00000246
Apr 24 00:43:27 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fb62b980 RCX: 0000000000f10000
Apr 24 00:43:27 fir-md1-s1 kernel: RDX: ffff8b533f81b780 RSI: 0000000001090101 RDI: ffff8b7339b45480
Apr 24 00:43:27 fir-md1-s1 kernel: RBP: ffff8b7321177800 R08: ffff8b633f7db780 R09: 0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: R10: ffff8b633f7df140 R11: fffff658360b8e00 R12: 0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: R13: ffff8b73211777a0 R14: ffff8b48fb62b6f0 R15: 0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: FS: 00007f35da56d880(0000) GS:ffff8b633f7c0000(0000) knlGS:0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 00:43:27 fir-md1-s1 kernel: CR2: 00007f35c9809ff0 CR3: 0000000dd2c10000 CR4: 00000000003407e0
Apr 24 00:43:27 fir-md1-s1 kernel: Call Trace:
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb5bfcb>] queued_spin_lock_slowpath+0xb/0xf
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb6a480>] _raw_spin_lock+0x20/0x30
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc11f1507>] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc1204d35>] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0d19632>] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc11c0518>] ldiskfs_map_blocks+0x98/0x700 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b501052>] ? ktime_get_ts64+0x52/0xf0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc12f6443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc12f8306>] osd_write_prep+0x2b6/0x360 [osd_ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc14a63d7>] mdt_obd_preprw+0x637/0x1060 [mdt]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e144be>] tgt_brw_write+0xc7e/0x1a90 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e0617d>] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e1810a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dbc6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0db9565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4d67c2>] ? default_wake_function+0x12/0x20
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4cba9b>] ? __wake_up_common+0x5b/0x90
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dc000c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dbf510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1c31>] kthread+0xd1/0xe0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb74c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:27 fir-md1-s1 kernel: Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 14 9c 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b
Apr 24 00:43:27 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#33 stuck for 22s! [mdt_io01_111:22207]
Apr 24 00:43:27 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif
Apr 24 00:43:27 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas
Apr 24 00:43:27 fir-md1-s1 kernel: CPU: 33 PID: 22207 Comm: mdt_io01_111 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1
Apr 24 00:43:27 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018
Apr 24 00:43:27 fir-md1-s1 kernel: task: ffff8b7322a74100 ti: ffff8b7322a9c000 task.ti: ffff8b7322a9c000
Apr 24 00:43:27 fir-md1-s1 kernel: RIP: 0010:[<ffffffff9b5121e6>] [<ffffffff9b5121e6>] native_queued_spin_lock_slowpath+0x126/0x200
Apr 24 00:43:27 fir-md1-s1 kernel: RSP: 0018:ffff8b7322a9f800 EFLAGS: 00000246
Apr 24 00:43:27 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fb62bda8 RCX: 0000000001090000
Apr 24 00:43:27 fir-md1-s1 kernel: RDX: ffff8b533f6db780 RSI: 0000000000690101 RDI: ffff8b7339b45480
Apr 24 00:43:27 fir-md1-s1 kernel: RBP: ffff8b7322a9f800 R08: ffff8b533f81b780 R09: 0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: R10: ffff8b533f81f140 R11: fffff657d8defe00 R12: 0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: R13: ffff8b7322a9f7a0 R14: ffff8b48fb62bb18 R15: 0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: FS: 00007fa109a46740(0000) GS:ffff8b533f800000(0000) knlGS:0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 00:43:27 fir-md1-s1 kernel: CR2: 00007fa1096331cc CR3: 0000000dd2c10000 CR4: 00000000003407e0
Apr 24 00:43:27 fir-md1-s1 kernel: Call Trace:
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb5bfcb>] queued_spin_lock_slowpath+0xb/0xf
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb6a480>] _raw_spin_lock+0x20/0x30
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc11f1507>] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc1204d35>] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0d19632>] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc11c0518>] ldiskfs_map_blocks+0x98/0x700 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b501052>] ? ktime_get_ts64+0x52/0xf0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc12f6443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc12f8306>] osd_write_prep+0x2b6/0x360 [osd_ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc14a63d7>] mdt_obd_preprw+0x637/0x1060 [mdt]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e144be>] tgt_brw_write+0xc7e/0x1a90 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4e011c>] ? update_curr+0x14c/0x1e0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e0617d>] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e1810a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dbc6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0db9565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4d67c2>] ? default_wake_function+0x12/0x20
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4cba9b>] ? __wake_up_common+0x5b/0x90
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dc000c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dbf510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1c31>] kthread+0xd1/0xe0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb74c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:27 fir-md1-s1 kernel: Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 14 9c 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2
Apr 24 00:43:27 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#36 stuck for 22s! [mdt_io00_018:21579]
Apr 24 00:43:27 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif
Apr 24 00:43:27 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas
Apr 24 00:43:27 fir-md1-s1 kernel: CPU: 36 PID: 21579 Comm: mdt_io00_018 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1
Apr 24 00:43:27 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018
Apr 24 00:43:27 fir-md1-s1 kernel: task: ffff8b733f0ce180 ti: ffff8b7322144000 task.ti: ffff8b7322144000
Apr 24 00:43:27 fir-md1-s1 kernel: RIP: 0010:[<ffffffff9b5121e8>] [<ffffffff9b5121e8>] native_queued_spin_lock_slowpath+0x128/0x200
Apr 24 00:43:27 fir-md1-s1 kernel: RSP: 0018:ffff8b7322147750 EFLAGS: 00000246
Apr 24 00:43:27 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b5d6cc4f3b0 RCX: 0000000001210000
Apr 24 00:43:27 fir-md1-s1 kernel: RDX: ffff8b433ee1b780 RSI: 0000000000010101 RDI: ffff8b7339b45480
Apr 24 00:43:27 fir-md1-s1 kernel: RBP: ffff8b7322147750 R08: ffff8b433f05b780 R09: 0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: R10: ffff8b433f05f140 R11: fffff6579cf19600 R12: 0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: R13: ffff8b73221476f0 R14: ffff8b5d6cc4f120 R15: 0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: FS: 00007f0c659f7740(0000) GS:ffff8b433f040000(0000) knlGS:0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 00:43:27 fir-md1-s1 kernel: CR2: 00007f0c655e41cc CR3: 0000000dd2c10000 CR4: 00000000003407e0
Apr 24 00:43:27 fir-md1-s1 kernel: Call Trace:
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb5bfcb>] queued_spin_lock_slowpath+0xb/0xf
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb6a480>] _raw_spin_lock+0x20/0x30
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc11f1507>] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc1204d35>] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b5d82c8>] ? zone_statistics+0x88/0xa0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc128a1a1>] ? qsd_op_begin+0xb1/0x4b0 [lquota]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc11c0518>] ldiskfs_map_blocks+0x98/0x700 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc11beb85>] ? ldiskfs_inode_attach_jinode+0x55/0xd0 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc12f6443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc12f7b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc11e62c9>] ? __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc14a5438>] mdt_commitrw_write.isra.46+0x608/0xd20 [mdt]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc14a717b>] mdt_obd_commitrw+0x29b/0x520 [mdt]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e11bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e1484d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4e011c>] ? update_curr+0x14c/0x1e0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0d69890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e1810a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dbc6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0db9565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4d67c2>] ? default_wake_function+0x12/0x20
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4cba9b>] ? __wake_up_common+0x5b/0x90
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dc000c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dbf510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1c31>] kthread+0xd1/0xe0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb74c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:27 fir-md1-s1 kernel: Code: 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 14 9c 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 85 c0 <74> f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0
Apr 24 00:43:27 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#42 stuck for 22s! [mdt_io02_025:21618]
Apr 24 00:43:27 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif
Apr 24 00:43:27 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas
Apr 24 00:43:27 fir-md1-s1 kernel: CPU: 42 PID: 21618 Comm: mdt_io02_025 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1
Apr 24 00:43:27 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018
Apr 24 00:43:27 fir-md1-s1 kernel: task: ffff8b732113c100 ti: ffff8b73273c8000 task.ti: ffff8b73273c8000
Apr 24 00:43:27 fir-md1-s1 kernel: RIP: 0010:[<ffffffff9b5121e2>] [<ffffffff9b5121e2>] native_queued_spin_lock_slowpath+0x122/0x200
Apr 24 00:43:27 fir-md1-s1 kernel: RSP: 0018:ffff8b73273cb800 EFLAGS: 00000246
Apr 24 00:43:27 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fb62ca20 RCX: 0000000001510000
Apr 24 00:43:27 fir-md1-s1 kernel: RDX: ffff8b633f7db780 RSI: 0000000000f10101 RDI: ffff8b7339b45480
Apr 24 00:43:27 fir-md1-s1 kernel: RBP: ffff8b73273cb800 R08: ffff8b633f89b780 R09: 0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: R10: ffff8b633f89f140 R11: fffff658299b9c00 R12: 0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: R13: ffff8b73273cb7a0 R14: ffff8b48fb62c790 R15: 0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: FS: 00007f35da56d880(0000) GS:ffff8b633f880000(0000) knlGS:0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 00:43:27 fir-md1-s1 kernel: CR2: 00007f35ca9b1ff0 CR3: 0000000dd2c10000 CR4: 00000000003407e0
Apr 24 00:43:27 fir-md1-s1 kernel: Call Trace:
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb5bfcb>] queued_spin_lock_slowpath+0xb/0xf
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb6a480>] _raw_spin_lock+0x20/0x30
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc11f1507>] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc1204d35>] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0d19632>] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc11c0518>] ldiskfs_map_blocks+0x98/0x700 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b501052>] ? ktime_get_ts64+0x52/0xf0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc12f6443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc12f8306>] osd_write_prep+0x2b6/0x360 [osd_ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc14a63d7>] mdt_obd_preprw+0x637/0x1060 [mdt]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e144be>] tgt_brw_write+0xc7e/0x1a90 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4e011c>] ? update_curr+0x14c/0x1e0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e0617d>] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e1810a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dbc6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0db9565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4d67c2>] ? default_wake_function+0x12/0x20
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4cba9b>] ? __wake_up_common+0x5b/0x90
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dc000c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dbf510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1c31>] kthread+0xd1/0xe0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb74c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:27 fir-md1-s1 kernel: Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 14 9c 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b
Apr 24 00:43:27 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#44 stuck for 22s! [mdt_io00_065:22104]
Apr 24 00:43:27 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif
Apr 24 00:43:27 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas
Apr 24 00:43:27 fir-md1-s1 kernel: CPU: 44 PID: 22104 Comm: mdt_io00_065 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1
Apr 24 00:43:27 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018
Apr 24 00:43:27 fir-md1-s1 kernel: task: ffff8b733c794100 ti: ffff8b5310244000 task.ti: ffff8b5310244000
Apr 24 00:43:27 fir-md1-s1 kernel: RIP: 0010:[<ffffffff9b5121e8>] [<ffffffff9b5121e8>] native_queued_spin_lock_slowpath+0x128/0x200
Apr 24 00:43:27 fir-md1-s1 kernel: RSP: 0018:ffff8b5310247800 EFLAGS: 00000246
Apr 24 00:43:27 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fb62c5f8 RCX: 0000000001610000
Apr 24 00:43:27 fir-md1-s1 kernel: RDX: ffff8b633f89b780 RSI: 0000000001510101 RDI: ffff8b7339b45480
Apr 24 00:43:27 fir-md1-s1 kernel: RBP: ffff8b5310247800 R08: ffff8b433f0db780 R09: 0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: R10: ffff8b433f0df140 R11: fffff65795efe400 R12: 0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: R13: ffff8b53102477a0 R14: ffff8b48fb62c368 R15: 0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: FS: 00007fa2ad0f3700(0000) GS:ffff8b433f0c0000(0000) knlGS:0000000000000000
Apr 24 00:43:27 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 00:43:27 fir-md1-s1 kernel: CR2: 00007efe23520000 CR3: 0000000dd2c10000 CR4: 00000000003407e0
Apr 24 00:43:27 fir-md1-s1 kernel: Call Trace:
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb5bfcb>] queued_spin_lock_slowpath+0xb/0xf
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb6a480>] _raw_spin_lock+0x20/0x30
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc11f1507>] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc1204d35>] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b61a849>] ? ___slab_alloc+0x209/0x4f0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0d19632>] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc11c0518>] ldiskfs_map_blocks+0x98/0x700 [ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b501052>] ? ktime_get_ts64+0x52/0xf0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc12f6443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc12f8306>] osd_write_prep+0x2b6/0x360 [osd_ldiskfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc14a63d7>] mdt_obd_preprw+0x637/0x1060 [mdt]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e144be>] tgt_brw_write+0xc7e/0x1a90 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4e011c>] ? update_curr+0x14c/0x1e0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e0617d>] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0e1810a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dbc6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0db9565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4d67c2>] ? default_wake_function+0x12/0x20
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4cba9b>] ? __wake_up_common+0x5b/0x90
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dc000c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dbf510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1c31>] kthread+0xd1/0xe0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb74c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:27 fir-md1-s1 kernel: Code: 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 14 9c 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 85 c0 <74> f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4d67b0>] ? wake_up_state+0x20/0x20
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dc000c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffffc0dbf510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1c31>] kthread+0xd1/0xe0
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9bb74c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 24 00:43:27 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:27 fir-md1-s1 kernel: Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 14 9c 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b
Apr 24 00:43:31 fir-md1-s1 kernel: NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! [mdt_io01_006:21534]
Apr 24 00:43:31 fir-md1-s1 kernel: Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx4_en(OE) mlx4_ib(OE) mlx4_core(OE) dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dcdbas ses enclosure ipmi_si pcspkr dm_multipath ipmi_devintf dm_mod ccp sg ipmi_msghandler k10temp i2c_piix4 acpi_power_meter knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif
Apr 24 00:43:31 fir-md1-s1 kernel: crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea mlxfw(OE) sysfillrect devlink sysimgblt fb_sys_fops mlx_compat(OE) ahci ttm crct10dif_pclmul crct10dif_common libahci drm tg3 crc32c_intel ptp libata megaraid_sas drm_panel_orientation_quirks pps_core mpt3sas(OE) raid_class scsi_transport_sas
Apr 24 00:43:31 fir-md1-s1 kernel: CPU: 25 PID: 21534 Comm: mdt_io01_006 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1
Apr 24 00:43:31 fir-md1-s1 kernel: Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018
Apr 24 00:43:31 fir-md1-s1 kernel: task: ffff8b7320ff4100 ti: ffff8b7338b8c000 task.ti: ffff8b7338b8c000
Apr 24 00:43:31 fir-md1-s1 kernel: RIP: 0010:[<ffffffff9b5121e2>] [<ffffffff9b5121e2>] native_queued_spin_lock_slowpath+0x122/0x200
Apr 24 00:43:31 fir-md1-s1 kernel: RSP: 0018:ffff8b7338b8f800 EFLAGS: 00000246
Apr 24 00:43:31 fir-md1-s1 kernel: RAX: 0000000000000000 RBX: ffff8b48fce60ff0 RCX: 0000000000c90000
Apr 24 00:43:31 fir-md1-s1 kernel: RDX: ffff8b433f0db780 RSI: 0000000001610101 RDI: ffff8b7339b45480
Apr 24 00:43:31 fir-md1-s1 kernel: RBP: ffff8b7338b8f800 R08: ffff8b533f79b780 R09: 0000000000000000
Apr 24 00:43:31 fir-md1-s1 kernel: R10: ffff8b533f79f140 R11: fffff657d7e9a800 R12: 0000000000000000
Apr 24 00:43:31 fir-md1-s1 kernel: R13: ffff8b7338b8f7a0 R14: ffff8b48fce60d60 R15: 0000000000000000
Apr 24 00:43:31 fir-md1-s1 kernel: FS: 00007f43a2d5c700(0000) GS:ffff8b533f780000(0000) knlGS:0000000000000000
Apr 24 00:43:31 fir-md1-s1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 24 00:43:31 fir-md1-s1 kernel: CR2: 00007f43a2dcf000 CR3: 0000000dd2c10000 CR4: 00000000003407e0
Apr 24 00:43:31 fir-md1-s1 kernel: Call Trace:
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffff9bb5bfcb>] queued_spin_lock_slowpath+0xb/0xf
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffff9bb6a480>] _raw_spin_lock+0x20/0x30
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc11f1507>] ldiskfs_es_lru_add+0x57/0x90 [ldiskfs]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc1204d35>] ldiskfs_ext_map_blocks+0x7b5/0xf60 [ldiskfs]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffff9b782a14>] ? vsnprintf+0x234/0x6a0
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc11c0518>] ldiskfs_map_blocks+0x98/0x700 [ldiskfs]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc09f9a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffff9b501052>] ? ktime_get_ts64+0x52/0xf0
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc12f6443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc12f8306>] osd_write_prep+0x2b6/0x360 [osd_ldiskfs]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc14a63d7>] mdt_obd_preprw+0x637/0x1060 [mdt]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc0e144be>] tgt_brw_write+0xc7e/0x1a90 [ptlrpc]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc0e06408>] ? tgt_free_reply_data+0x128/0x3b0 [ptlrpc]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffff9b61beb6>] ? kfree+0x106/0x140
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc0e06408>] ? tgt_free_reply_data+0x128/0x3b0 [ptlrpc]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc0e0617d>] ? tgt_lookup_reply+0x2d/0x190 [ptlrpc]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc0e1810a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc09f9f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc0dbc6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc0db9565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffff9b4d67c2>] ? default_wake_function+0x12/0x20
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffff9b4cba9b>] ? __wake_up_common+0x5b/0x90
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc0dc000c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffffc0dbf510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffff9b4c1c31>] kthread+0xd1/0xe0
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffff9bb74c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 24 00:43:31 fir-md1-s1 kernel: [<ffffffff9b4c1b60>] ? insert_kthread_work+0x40/0x40
Apr 24 00:43:31 fir-md1-s1 kernel: Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 14 9c 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b
Apr 24 00:43:31 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client ac39260b-aa99-2050-abe9-06dcf51e927b (at 10.9.101.47@o2ib4) reconnecting
Apr 24 00:43:31 fir-md1-s1 kernel: Lustre: Skipped 258 previous similar messages
Apr 24 00:43:33 fir-md1-s1 kernel: sched: RT throttling activated
Apr 24 00:43:33 fir-md1-s1 kernel: Lustre: 48105:0:(service.c:2011:ptlrpc_server_handle_req_in()) @@@ Slow req_in handling 16s req@ffff8b5d60ac1c50 x1631573995472736/t0(0) o4->661f0cfa-e148-dc98-69cd-517192e597e7@10.8.7.3@o2ib6:0/0 lens 4216/0 e 0 to 0 dl 0 ref 1 fl New:/2/ffffffff rc 0/-1
Apr 24 00:43:33 fir-md1-s1 kernel: Lustre: 21579:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20:17s); client may timeout. req@ffff8b42da5c3050 x1630937895792336/t214891143440(0) o4->59bf854b-f25a-3380-6e0c-7f5c7b945dd1@10.9.108.15@o2ib4:15/0 lens 488/416 e 1 to 0 dl 1556091795 ref 1 fl Complete:/0/0 rc 0/0
Apr 24 00:43:33 fir-md1-s1 kernel: LustreError: 21594:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff8b53203bc050 x1631542737016096/t0(0) o3->be42b497-ab1b-8d58-3101-014aad577cfc@10.8.27.35@o2ib6:14/0 lens 488/440 e 1 to 0 dl 1556091794 ref 1 fl Interpret:/0/0 rc 0/0
Apr 24 00:43:33 fir-md1-s1 kernel: Lustre: fir-MDT0002: Bulk IO read error with be42b497-ab1b-8d58-3101-014aad577cfc (at 10.8.27.35@o2ib6), client will retry: rc -107
Apr 24 00:43:33 fir-md1-s1 kernel: Lustre: mdt_io: This server is not able to keep up with request traffic (cpu-bound).
Apr 24 00:43:33 fir-md1-s1 kernel: Lustre: 21690:0:(service.c:1541:ptlrpc_at_check_timed()) earlyQ=73 reqQ=0 recA=88, svcEst=20, delay=15960
Apr 24 00:43:33 fir-md1-s1 kernel: Lustre: 21690:0:(service.c:1322:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff8b5d5fab8850 x1631679054779200/t214891143471(0) o4->a82097ea-0a83-cc99-985b-882074216844@10.8.12.13@o2ib6:1/0 lens 3112/448 e 1 to 0 dl 1556091811 ref 2 fl Interpret:/0/0 rc 0/0
Apr 24 00:43:33 fir-md1-s1 kernel: LustreError: 21640:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.9.113.2@o2ib4: deadline 6:9s ago
req@ffff8b5d5f463050 x1631347528787136/t0(0) o4->8829dd2a-c714-d8f5-ab1a-941339d64495@10.9.113.2@o2ib4:23/0 lens 3544/0 e 0 to 0 dl 1556091803 ref 1 fl Interpret:/2/ffffffff rc 0/-1
Apr 24 00:43:33 fir-md1-s1 kernel: LustreError: 21610:0:(tgt_handler.c:644:process_req_last_xid()) @@@ Unexpected xid 5cbb2529bac50 vs. last_xid 5cbb2529bac5f
req@ffff8b48f8e9bc50 x1631341634104400/t0(0) o4->b984ccf1-6625-41c4-2ebc-91a79a8a0ab1@10.8.23.18@o2ib6:6/0 lens 2928/0 e 0 to 0 dl 1556091816 ref 1 fl Interpret:/2/ffffffff rc 0/-1
Apr 24 00:43:33 fir-md1-s1 kernel: Lustre: 48105:0:(service.c:2011:ptlrpc_server_handle_req_in()) Skipped 24 previous similar messages
Apr 24 00:45:09 fir-md1-s1 kernel: Lustre: fir-MDT0000: haven't heard from client cbbea467-025f-86a9-4e0e-ef1c851a3dc7 (at 10.8.26.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8b42ec506000, cur 1556091909 expire 1556091759 last 1556091682
Apr 24 00:45:09 fir-md1-s1 kernel: Lustre: Skipped 8 previous similar messages
Apr 24 00:48:45 fir-md1-s1 kernel: Lustre: 22086:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1556092118/real 1556092118] req@ffff8b5db3396300 x1631585366495376/t0(0) o104->fir-MDT0002@10.8.12.31@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1556092125 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Apr 24 00:48:45 fir-md1-s1 kernel: Lustre: 22086:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 23 previous similar messages
Apr 24 00:49:03 fir-md1-s1 kernel: Lustre: 21734:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply
req@ffff8b620dba6c00 x1631532464055968/t0(0) o101->39fe18a4-a89c-1a84-3eb2-1fc3124ee4a0@10.9.108.28@o2ib4:8/0 lens 480/568 e 0 to 0 dl 1556092148 ref 2 fl Interpret:/0/0 rc 0/0
Apr 24 00:49:03 fir-md1-s1 kernel: Lustre: 21734:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 509 previous similar messages
Apr 24 00:49:13 fir-md1-s1 kernel: LustreError: 22086:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.12.31@o2ib6) failed to reply to blocking AST (req@ffff8b5db3396300 x1631585366495376 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8b523c68b3c0/0x378007f0e0e192a1 lrc: 4/0,0 mode: PR/PR res: [0x2c001a5b3:0x636:0x0].0x0 bits 0x5b/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.8.12.31@o2ib6 remote: 0xbbfac8fcd3f5cadd expref: 3748 pid: 21505 timeout: 92010 lvb_type: 0
Apr 24 00:49:13 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.12.31@o2ib6 was evicted due to a lock blocking callback time out: rc -110
Apr 24 00:49:13 fir-md1-s1 kernel: LustreError: 20444:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 35s: evicting client at 10.8.12.31@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8b523c68b3c0/0x378007f0e0e192a1 lrc: 3/0,0 mode: PR/PR res: [0x2c001a5b3:0x636:0x0].0x0 bits 0x5b/0x0 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.8.12.31@o2ib6 remote: 0xbbfac8fcd3f5cadd expref: 3749 pid: 21505 timeout: 0 lvb_type: 0
Apr 24 00:51:29 fir-md1-s1 kernel: Lustre: 22086:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1556092282/real 1556092282] req@ffff8b5dbb385100 x1631585368677216/t0(0) o104->fir-MDT0002@10.8.22.1@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1556092289 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Apr 24 00:51:29 fir-md1-s1 kernel: Lustre: 22086:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Apr 24 00:51:47 fir-md1-s1 kernel: Lustre: 22021:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-5), not sending early reply
req@ffff8b628e6d6900 x1631535005686928/t214891595924(0) o36->6ea10cfc-48e7-f6e7-b834-4eb6674e3061@10.9.102.48@o2ib4:22/0 lens 488/3152 e 0 to 0 dl 1556092312 ref 2 fl Interpret:/0/0 rc 0/0
Apr 24 00:51:57 fir-md1-s1 kernel: LustreError: 22086:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.22.1@o2ib6) failed to reply to blocking AST (req@ffff8b5dbb385100 x1631585368677216 status 0 rc -110), evict it ns: mdt-fir-MDT0002_UUID lock: ffff8b52a68557c0/0x378007f0e0fed702 lrc: 4/0,0 mode: PR/PR res: [0x2c001a748:0x1db:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.22.1@o2ib6 remote: 0x5a75d9a873fe4860 expref: 3715 pid: 21705 timeout: 92173 lvb_type: 0
Apr 24 00:51:57 fir-md1-s1 kernel: LustreError: 138-a: fir-MDT0002: A client on nid 10.8.22.1@o2ib6 was evicted due to a lock blocking callback time out: rc -110
Apr 24 00:51:57 fir-md1-s1 kernel: LustreError: 20444:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 35s: evicting client at 10.8.22.1@o2ib6 ns: mdt-fir-MDT0002_UUID lock: ffff8b52a68557c0/0x378007f0e0fed702 lrc: 3/0,0 mode: PR/PR res: [0x2c001a748:0x1db:0x0].0x0 bits 0x5b/0x0 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.22.1@o2ib6 remote: 0x5a75d9a873fe4860 expref: 3716 pid: 21705 timeout: 0 lvb_type: 0
Apr 24 00:52:26 fir-md1-s1 kernel: Lustre: fir-MDT0002: Connection restored to ef4ceb90-5603-14d7-21e5-f4e7d223a0a1 (at 10.8.8.33@o2ib6)
Apr 24 00:52:26 fir-md1-s1 kernel: Lustre: Skipped 287 previous similar messages
Apr 24 00:53:50 fir-md1-s1 kernel: Lustre: fir-MDT0002: Client f50d7e88-e046-b55a-0fe3-13cbe95d417b (at 10.8.8.33@o2ib6) reconnecting