Details
-
Bug
-
Resolution: Won't Fix
-
Major
-
None
-
Lustre 2.9.0
-
None
-
Centos 6.8
-
3
-
9223372036854775807
Description
MDS suddenly stopped working, unknown cause of it, we get the following trace.
CentOS 6.8
Versions:
kernel-2.6.32-573.12.1.el6_lustre.x86_64
kernel-devel-2.6.32-573.12.1.el6_lustre.x86_64
kernel-firmware-2.6.32-573.12.1.el6_lustre.x86_64
kmod-lustre-2.9.0-1.el6.x86_64
kmod-lustre-osd-ldiskfs-2.9.0-1.el6.x86_64
kmod-lustre-osd-zfs-2.9.0-1.el6.x86_64
lustre-2.9.0-1.el6.x86_64
lustre-iokit-2.9.0-1.el6.x86_64
lustre-osd-ldiskfs-mount-2.9.0-1.el6.x86_64
lustre-osd-zfs-mount-2.9.0-1.el6.x86_64
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: BUG: unable to handle kernel paging request at 0000000240004813
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: IP: [<ffffffffa08b5032>] lu_obj_hop_keycmp+0x12/0x20 [obdclass]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: PGD 0
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: Oops: 0000 1 SMP
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: last sysfs file: /sys/devices/system/cpu/online
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: CPU 11
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgs(U) mgc(U) vfat fat usb_storage osd_ldiskfs(U) ldiskfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) mpt3sas mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu 8021q garp stp llc drbd(U) libcrc32c nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) microcode iTCO_wdt iTCO_vendor_support dcdbas ipmi_devintf power_meter acpi_ipmi ipmi_si ipmi_msghandler joydev sg shpchp igb i2c_algo_bit i2c_core ixgbe dca mdio sb_edac edac_core lpc_ich mfd_core ext4 jbd2 mbcache sd_mod crc_t10dif mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en ptp pps_core mlx4_core ahci megaraid_sas wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel:
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: Pid: 11076, comm: mdt01_005 Tainted: P - ------------ 2.6.32-573.12.1.el6_lustre.x86_64 #1 Dell Inc. PowerEdge R730xd/072T6D
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: RIP: 0010:[<ffffffffa08b5032>] [<ffffffffa08b5032>] lu_obj_hop_keycmp+0x12/0x20 [obdclass]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: RSP: 0018:ffff8807c7eff8e0 EFLAGS: 00010202
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: RAX: ffffffffa090d240 RBX: ffff881002938240 RCX: 0000000000000010
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: RDX: 0000000000000000 RSI: 0000000240004813 RDI: ffff88105b649308
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: RBP: ffff8807c7eff8e0 R08: 0000000000000001 R09: 00000000000068ac
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: R10: 000000000000001b R11: 9000000000000000 R12: ffff8807c7eff9e0
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: R13: ffff88105b649308 R14: 0000000000000000 R15: 0000000240004833
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: FS: 0000000000000000(0000) GS:ffff88089c4a0000(0000) knlGS:0000000000000000
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: CR2: 0000000240004813 CR3: 0000000001a8d000 CR4: 00000000001407e0
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: Process mdt01_005 (pid: 11076, threadinfo ffff8807c7efc000, task ffff88086edd5520)
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: Stack:
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: ffff8807c7eff930 ffffffffa0794f25 0000000100000000 0000000000000000
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: <d> ffff8807c7eff910 ffff88081bfe8098 ffff8807c7eff9e0 0000000000000030
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: <d> 0000000000000010 ffffc90046adf000 ffff8807c7eff940 ffffffffa07950f6
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: Call Trace:
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa0794f25>] cfs_hash_bd_lookup_intent+0x65/0x130 [libcfs]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa07950f6>] cfs_hash_bd_peek_locked+0x16/0x20 [libcfs]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa08b6817>] htable_lookup+0x77/0x210 [obdclass]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa08b784b>] lu_object_find_try+0x8b/0x260 [obdclass]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa0aafdb6>] ? lustre_msg_string+0x96/0x290 [ptlrpc]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa08b7ad1>] lu_object_find_at+0xb1/0xe0 [obdclass]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa0aafd15>] ? lustre_msg_buf+0x55/0x60 [ptlrpc]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa0ad7402>] ? __req_capsule_get+0x162/0x6e0 [ptlrpc]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa08b7b16>] lu_object_find+0x16/0x20 [obdclass]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa10eead6>] mdt_object_find+0x56/0x170 [mdt]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa11030df>] mdt_getattr_name_lock+0xb1f/0x1910 [mdt]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa0ab26a4>] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa11043f2>] mdt_intent_getattr+0x292/0x470 [mdt]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa10f491e>] mdt_intent_policy+0x4be/0xc70 [mdt]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa0a627c7>] ldlm_lock_enqueue+0x127/0x990 [ptlrpc]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa0a8e17b>] ldlm_handle_enqueue0+0x98b/0x14e0 [ptlrpc]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa0b02cd1>] ? tgt_lookup_reply+0x31/0x190 [ptlrpc]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa0b142d1>] tgt_enqueue+0x61/0x230 [ptlrpc]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa0b1547c>] tgt_request_handle+0x8ec/0x1440 [ptlrpc]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa0ac1b91>] ptlrpc_main+0xd31/0x1800 [ptlrpc]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffff815391be>] ? thread_return+0x4e/0x7d0
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffffa0ac0e60>] ? ptlrpc_main+0x0/0x1800 [ptlrpc]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffff810a0fce>] kthread+0x9e/0xc0
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffff8100c28a>] child_rip+0xa/0x20
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffff810a0f30>] ? kthread+0x0/0xc0
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: [<ffffffff8100c280>] ? child_rip+0x0/0x20
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: Code: 83 7b 68 00 74 09 48 8d 7b 68 e8 5a a4 fd ff 48 83 c4 08 5b c9 c3 0f 1f 00 55 48 89 e5 0f 1f 44 00 00 b9 10 00 00 00 48 83 ee 20 <f3> a6 c9 0f 94 c0 0f b6 c0 c3 0f 1f 40 00 55 48 89 e5 48 83 ec
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: RIP [<ffffffffa08b5032>] lu_obj_hop_keycmp+0x12/0x20 [obdclass]
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: RSP <ffff8807c7eff8e0>
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: CR2: 0000000240004813
Jan 13 02:00:19 LustreB-MDS-bzp9rg2 kernel: --[ end trace 57c83a9477656035 ]--