[LU-11781] MDS hit BUG: unable to handle kernel NULL pointer dereference

Duplicate — Sat, 15 Dec 2018 00:56:45 +0000
MDS 0 hit kernel panic after running about 24 hours.
MDS 0 console
[18476.473668] Lustre: MGS: haven't heard from client 8e54524b-a52c-7091-0e06-f2d4a89dd59c (at 192.168.1.109@o2ib) in 227 seconds. I think it's dead, and I am evic
ting it. exp ffff9fcae4159400, cur 1544788503 expire 1544788353 last 1544788276
[18520.421813] LNet: 28111:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 192.168.1.109@o2ib: 0 seconds
[18520.433290] LNet: 28111:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 1 previous similar message
[18521.243163] LustreError: 137-5: soaked-MDT0001_UUID: not available for connect from 192.168.1.126@o2ib (no target). If you are running an HA pair check that the
 target is mounted on the other server.
[18521.263041] LustreError: Skipped 122 previous similar messages
[18571.421594] LNet: 28111:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 192.168.1.109@o2ib: 1 seconds
[18620.421324] LNet: 28111:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 192.168.1.109@o2ib: 0 seconds
[18703.424361] Lustre: MGS: Connection restored to 192.168.1.109@o2ib (at 192.168.1.109@o2ib)
[18704.407564] Lustre: soaked-MDT0000: Received new LWP connection from 192.168.1.109@o2ib, removing former export from same NID
[18737.363675] Lustre: soaked-MDT0000: Received new LWP connection from 192.168.1.110@o2ib, removing former export from same NID
[18737.376324] Lustre: Skipped 1 previous similar message
[18737.382130] Lustre: soaked-MDT0000: Connection restored to 192.168.1.110@o2ib (at 192.168.1.110@o2ib)
[18737.392457] Lustre: Skipped 2 previous similar messages
[18737.423007] LustreError: 31941:0:(osd_oi.c:761:osd_oi_insert()) dm-2: the FID [0x20000c768:0x179ce:0x0] is used by two objects: 402128901/3006680378 357564421/3
006680381
[18737.440029] LNet: 28116:0:(o2iblnd_cb.c:408:kiblnd_handle_rx()) PUT_NACK from 192.168.1.110@o2ib
[18741.123580] LustreError: 167-0: soaked-MDT0001-osp-MDT0000: This client was evicted by soaked-MDT0001; in progress operations using this service will fail.
[18753.014392] BUG: unable to handle kernel NULL pointer dereference at           (null)
[18753.023170] IP: [<          (null)>]           (null)
[18753.028819] PGD 0 
[18753.031075] Oops: 0010 [#1] SMP 
[18753.034700] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ko2iblnd(OE) ptl
rpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_c
m(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) mlx4_ib(OE) ib_core(OE) dm_round_robin sb_edac intel_powerclamp coretemp intel_rapl
 iosf_mbi kvm iTCO_wdt irqbypass iTCO_vendor_support sg crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr ipmi
_ssif mei_me mei lpc_ich wmi i2c_i801 ioatdma ipmi_si ipmi_devintf ipmi_msghandler dm_multipath dm_mod auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t1
0dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops isci igb ttm mpt2sas ahci crct10dif_pclmul crct10dif_common libsas liba
hci crc32c_intel ptp drm mlx4_core(OE) raid_class libata pps_core drm_panel_orientation_quirks scsi_transport_sas mlx_compat(OE) dca devlink i2c_algo_bit
[18753.146014] CPU: 0 PID: 43000 Comm: mdt_out00_019 Kdump: loaded Tainted: G           OE  ------------   3.10.0-957.el7_lustre.x86_64 #1
[18753.159604] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013
[18753.172121] task: ffff9fcf2591e180 ti: ffff9fcad0a8c000 task.ti: ffff9fcad0a8c000
[18753.180476] RIP: 0010:[<0000000000000000>]  [<          (null)>]           (null)
[18753.188859] RSP: 0018:ffff9fcad0a8fb60  EFLAGS: 00010246
[18753.194788] RAX: 0000000000000000 RBX: ffff9fc7a8190000 RCX: 0000000000000002
[18753.202754] RDX: ffffffffc12dc770 RSI: ffff9fcad0a8fb68 RDI: ffff9fc7a8190008
[18753.210718] RBP: ffff9fcad0a8fba0 R08: 0000000000000004 R09: 0000000000000000
[18753.218682] R10: 0000000000000001 R11: 00000000007fffff R12: ffff9fc736087300
[18753.226645] R13: ffff9fcb00494c48 R14: ffff9fcefba3a200 R15: ffff9fc7a8190008
[18753.234611] FS:  0000000000000000(0000) GS:ffff9fcb2e000000(0000) knlGS:0000000000000000
[18753.243647] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[18753.250052] CR2: 0000000000000000 CR3: 0000000551a10000 CR4: 00000000000607f0
[18753.258016] Call Trace:
[18753.260766]  [<ffffffffc12dabee>] ? osd_ldiskfs_it_fill+0xbe/0x260 [osd_ldiskfs]
[18753.269028]  [<ffffffffc12dadc7>] osd_it_ea_load+0x37/0x100 [osd_ldiskfs]
[18753.276640]  [<ffffffffc0e84038>] dt_index_walk+0xf8/0x430 [obdclass]
[18753.283850]  [<ffffffffc0e84370>] ? dt_index_walk+0x430/0x430 [obdclass]
[18753.291350]  [<ffffffffc0e85444>] dt_index_read+0x394/0x6a0 [obdclass]
[18753.298701]  [<ffffffffc10ceb32>] tgt_obd_idx_read+0x612/0x860 [ptlrpc]
[18753.306117]  [<ffffffffc10d135a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
[18753.313825]  [<ffffffffc10aaa51>] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc]
[18753.322284]  [<ffffffffc0bdebde>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
[18753.330186]  [<ffffffffc107592b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[18753.338795]  [<ffffffffc10727b5>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
[18753.346380]  [<ffffffff9fcd67c2>] ? default_wake_function+0x12/0x20
[18753.353379]  [<ffffffff9fccba9b>] ? __wake_up_common+0x5b/0x90
[18753.359934]  [<ffffffffc107925c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
[18753.366951]  [<ffffffffc1078760>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
[18753.375209]  [<ffffffff9fcc1c31>] kthread+0xd1/0xe0
[18753.380662]  [<ffffffff9fcc1b60>] ? insert_kthread_work+0x40/0x40
[18753.387480]  [<ffffffffa0374c37>] ret_from_fork_nospec_begin+0x21/0x21
[18753.394766]  [<ffffffff9fcc1b60>] ? insert_kthread_work+0x40/0x40
[18753.401564] Code:  Bad RIP value.
[18753.405279] RIP  [<          (null)>]           (null)
[18753.411023]  RSP <ffff9fcad0a8fb60>
[18753.414916] CR2: 0000000000000000
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.10.0-957.el7_lustre.x86_64 (jenkins@trevis-309-el7-x8664-2.trevis.whamcloud.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Sat Dec 8 05:53:16 UTC 2018
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.el7_lustre.x86_64 ro console=ttyS0,115200 irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr disable_cpu_apicid=0 elfcorehdr=869816K
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000000fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000001000-0x000000000008efff] usable
[    0.000000] BIOS-e820: [mem 0x000000000008f000-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x000000002b000000-0x000000003516dfff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bb3c7000-0x00000000bdd2efff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000bdd2f000-0x00000000bddccfff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000bddcd000-0x00000000bdea0fff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000bdea1000-0x00000000bdf2efff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x00000000bdf2f000-0x00000000bdfabfff] ACPI data
[    0.000000] BIOS-e820: [mem 0x00000000be000000-0x00000000cfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed19000-0x00000000fed19fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ffa20000-0x00000000ffffffff] reserved
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.6 present.
Whamcloud Community JIRA

[LU-11781] MDS hit BUG: unable to handle kernel NULL pointer dereference
