[67171.114253] Lustre: Lustre: Build Version: 2.12.0_RC2_1_g3a78e96 [67171.237817] LNet: Using FastReg for registration [67171.372648] LNet: Added LNI 10.0.10.51@o2ib7 [8/256/0/180] [67172.236616] LNet: 114356:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.202@o2ib7: 67169 seconds [67172.502599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [67172.815820] Lustre: MGS: Connection restored to MGC10.0.10.51@o2ib7_0 (at 0@lo) [67176.868507] Lustre: MGS: Connection restored to 45b317c9-9bf8-d2d7-dd4b-6df9dc8479bd (at 10.9.101.60@o2ib4) [67195.110347] Lustre: MGS: Connection restored to 541753c6-9032-f329-82ff-d9c72bbb1c24 (at 10.0.10.105@o2ib7) [67195.120094] Lustre: Skipped 1 previous similar message [67197.126683] Lustre: MGS: Connection restored to 777fb8a1-99a3-6ac6-5b75-2178343045db (at 10.0.10.108@o2ib7) [67204.860885] Lustre: MGS: Connection restored to d1fa3f21-b7f3-5a11-e34d-8236124081cd (at 10.9.0.1@o2ib4) [67204.870370] Lustre: Skipped 1 previous similar message [67213.116967] Lustre: MGS: Connection restored to 73368e2a-0b82-5939-f05f-1ad2a79768da (at 10.9.101.59@o2ib4) [67213.126710] Lustre: Skipped 8 previous similar messages [67248.704669] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5 [67248.711631] LDISKFS-fs (dm-1): file extents enabled, maximum tree depth=5 [67248.970369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [67248.974610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [67250.619118] LustreError: 137-5: fir-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [67250.664135] LustreError: 11-0: fir-OST0013-osc-MDT0002: operation ost_connect to node 10.0.10.104@o2ib7 failed: rc = -16 [67250.682539] Lustre: fir-MDT0002: Imperative Recovery not enabled, recovery window 300-900 [67250.692215] Lustre: fir-MDT0002: in recovery but waiting for the first client to connect [67250.958265] Lustre: fir-MDT0002: Will be in recovery for at least 5:00, or until 16 clients reconnect [67251.138788] Lustre: fir-MDT0002: Connection restored to 10.0.10.107@o2ib7 (at 10.0.10.107@o2ib7) [67251.139111] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 10.0.10.107@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [67251.139113] LustreError: Skipped 1 previous similar message [67251.170509] Lustre: Skipped 12 previous similar messages [67256.185533] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.0.10.105@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [67256.202907] LustreError: Skipped 23 previous similar messages [67262.120702] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.0.10.102@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [67262.138072] LustreError: Skipped 26 previous similar messages [67267.131183] LustreError: 137-5: fir-MDT0003_UUID: not available for connect from 10.0.10.104@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [67267.148596] LustreError: Skipped 6 previous similar messages [67275.836348] LustreError: 137-5: fir-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [67275.852606] LustreError: Skipped 10 previous similar messages [67287.145207] Lustre: fir-MDT0002: Connection restored to 10.0.10.102@o2ib7 (at 10.0.10.102@o2ib7) [67287.154003] Lustre: Skipped 90 previous similar messages [67301.824418] Lustre: fir-MDT0002: Recovery over after 0:51, of 16 clients 16 recovered and 0 were evicted. [67301.833988] Lustre: Skipped 1 previous similar message [67589.433927] BUG: unable to handle kernel paging request at ffff886b3ab2e000 [67589.440942] IP: [] osd_it_ea_rec+0x2f3/0x610 [osd_ldiskfs] [67589.448032] PGD 925452067 PUD 203c240063 PMD 203b28c063 PTE 800000203ab2e061 [67589.455182] Oops: 0003 [#1] SMP [67589.458468] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) osc(OE) lov(OE) fid(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ib_ucm rpcrdma rdma_ucm ib_uverbs ib_iser ib_umad rdma_cm iw_cm libiscsi ib_ipoib scsi_transport_iscsi ib_cm mlx5_ib ib_core mpt2sas mptctl mptbase dell_rbu sunrpc vfat fat dm_round_robin amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ses ghash_clmulni_intel aesni_intel dcdbas lrw gf128mul glue_helper ablk_helper enclosure cryptd ipmi_si pcspkr dm_multipath ipmi_devintf ccp k10temp sg i2c_piix4 ipmi_msghandler dm_mod acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif [67589.531510] crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mlx5_core ttm ahci mlxfw libahci drm devlink crct10dif_pclmul tg3 crct10dif_common crc32c_intel libata megaraid_sas drm_panel_orientation_quirks ptp pps_core mpt3sas(OE) raid_class scsi_transport_sas [last unloaded: libcfs] [67589.559803] CPU: 13 PID: 114706 Comm: mdt_out01_000 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [67589.572480] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.3.6 04/20/2018 [67589.580046] task: ffff885b338830c0 ti: ffff885b19078000 task.ti: ffff885b19078000 [67589.587525] RIP: 0010:[] [] osd_it_ea_rec+0x2f3/0x610 [osd_ldiskfs] [67589.597034] RSP: 0018:ffff885b1907bae8 EFLAGS: 00010246 [67589.602345] RAX: 0000000000000010 RBX: ffff886b3ab2dfd0 RCX: ffff886b3ab2e000 [67589.609478] RDX: 0000000000000010 RSI: ffff886b2fa3a6fe RDI: ffff886b3ab2dff0 [67589.616611] RBP: ffff885b1907bb40 R08: ffff886b3ab2e000 R09: 0000000000000018 [67589.623743] R10: ffff885a9d693a00 R11: ffff885a9d693a00 R12: ffff885b1cadc000 [67589.630876] R13: ffff886b2fa3a6c8 R14: 0000000000000000 R15: 0000000000000010 [67589.638009] FS: 00007f537271b900(0000) GS:ffff886b3f6c0000(0000) knlGS:0000000000000000 [67589.646097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [67589.651841] CR2: ffff886b3ab2e000 CR3: 000000403b9fe000 CR4: 00000000003407e0 [67589.658975] Call Trace: [67589.661457] [] dt_index_page_build+0x173/0x4e0 [obdclass] [67589.668520] [] dt_index_walk+0x1a0/0x430 [obdclass] [67589.675061] [] ? dt_index_walk+0x430/0x430 [obdclass] [67589.681777] [] dt_index_read+0x394/0x6a0 [obdclass] [67589.688349] [] tgt_obd_idx_read+0x612/0x860 [ptlrpc] [67589.694994] [] tgt_request_handle+0xaea/0x1580 [ptlrpc] [67589.701901] [] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [67589.709476] [] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [67589.716557] [] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [67589.724243] [] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [67589.731037] [] ? default_wake_function+0x12/0x20 [67589.737300] [] ? __wake_up_common+0x5b/0x90 [67589.743164] [] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [67589.749462] [] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [67589.756854] [] kthread+0xd1/0xe0 [67589.761732] [] ? insert_kthread_work+0x40/0x40 [67589.767826] [] ret_from_fork_nospec_begin+0xe/0x21 [67589.774262] [] ? insert_kthread_work+0x40/0x40 [67589.780354] Code: 20 74 0c 48 8d 42 21 48 83 e0 fe 48 83 c0 02 48 83 c0 07 48 8d 7b 20 48 83 e0 f8 66 89 43 18 e8 d4 bb 97 d5 41 0f b7 c7 48 63 d0 44 13 20 00 f6 43 1c 02 66 44 89 7b 1a 74 14 0f b7 55 c0 83 [67589.800876] RIP [] osd_it_ea_rec+0x2f3/0x610 [osd_ldiskfs] [67589.808044] RSP [67589.811539] CR2: ffff886b3ab2e000