[LU-7809] general protection fault: 0000 during failback of MDS disk resources

Related — Tue, 13 Sep 2016 20:05:26 +0000

Error happens during soak testing of build '20160222' (see https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20150222) DNE is enabled. MDTs had been formatted using ldiskfs, OST using zfs. MDSes are configured in active-active HA failover configuration.Especially nodes affected (lola-[8,9]) form a HA failover pair.
More set-up details can be found at https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-Configuration

Sequence of events

2016-02-23 23:52:17,963:fsmgmt.fsmgmt:INFO triggering fault mds_failover
2016-02-23 23:52:17,964:fsmgmt.fsmgmt:INFO reseting MDS node lola-9
2016-02-24 00:00:29 Both MDTs (mdt-2,3) failover to lola-8
2016-02-24 00:01:06,468:fsmgmt.fsmgmt:INFO ... soaked-MDT0003 failed back (action completed successful!)
2016-02-24 00:01:06,468:fsmgmt.fsmgmt:INFO Unmounting soaked-MDT0002 on lola-8 .. (--> caused crash)

The error reads as:

<4>general protection fault: 0000 [#1] 
<3>LustreError: 6683:0:(ldlm_lib.c:2562:target_stop_recovery_thread()) soaked-MDT0002: Aborting recovery
<4>SMP 
<4>last sysfs file: /sys/devices/system/cpu/online
<4>CPU 12 
<4>Modules linked in: mgs(U) osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) 8021q garp stp llc nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm scsi_dh_rdac dm_round_robin dm_multipath iTCO_wdt iTCO_vendor_support microcode zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core i2c_i801 ioatdma sg igb dca i2c_algo_bit i2c_core ptp pps_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas wmi mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
<4>
<4>Pid: 6617, comm: tgt_recover_2 Tainted: P           ---------------    2.6.32-504.30.3.el6_lustre.g93f956d.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ
<4>RIP: 0010:[<ffffffffa0b2222c>]  [<ffffffffa0b2222c>] distribute_txn_get_next_transno+0x3c/0xd0 [ptlrpc]
<4>RSP: 0018:ffff88028674fca0  EFLAGS: 00010207
<4>RAX: 5a5a5a5a5a5a5a5a RBX: 0000000000000000 RCX: 0000000000000000
<4>RDX: ffff8802866a41e8 RSI: ffffffffa0a65b80 RDI: ffff8802866a4208
<4>RBP: ffff88028674fcc0 R08: 00000000fffffff2 R09: 00000000fffffff5
<4>R10: 0000000000000009 R11: 0000000000000000 R12: ffff8802866a4180
<4>R13: ffff8802866a4208 R14: ffff8802866a4180 R15: 0000000000000000
<4>FS:  0000000000000000(0000) GS:ffff88044e480000(0000) knlGS:0000000000000000
<4>CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<4>CR2: 00007f7e64e29000 CR3: 0000000001a85000 CR4: 00000000000407e0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process tgt_recover_2 (pid: 6617, threadinfo ffff88028674e000, task ffff88028674d520)
<4>Stack:
<4> ffffffffa0bb6640 ffff88080c3fd038 0000000000000000 ffff88080c3fd3cc
<4><d> ffff88028674fd50 ffffffffa0a65c07 ffff88028674fde0 00000000a085c87e
<4><d> 0000000000000000 ffff8807fb3adddd 00000054a0a60fc0 ffffffffa0b52dc9
<4>Call Trace:
<4> [<ffffffffa0a65c07>] check_for_next_transno+0x87/0x6d0 [ptlrpc]
<4> [<ffffffffa0a65b80>] ? check_for_next_transno+0x0/0x6d0 [ptlrpc]
<4> [<ffffffffa0a62c63>] target_recovery_overseer+0xb3/0x630 [ptlrpc]
<4> [<ffffffffa0a60f30>] ? exp_req_replay_healthy_or_from_mdt+0x0/0x40 [ptlrpc]
<4> [<ffffffffa077bcf1>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
<4> [<ffffffffa0a62ac0>] ? abort_lock_replay_queue+0x30/0x120 [ptlrpc]
<4> [<ffffffffa0a693db>] target_recovery_thread+0x8bb/0x1dd0 [ptlrpc]
<4> [<ffffffff81064c12>] ? default_wake_function+0x12/0x20
<4> [<ffffffffa0a68b20>] ? target_recovery_thread+0x0/0x1dd0 [ptlrpc]
<4> [<ffffffff8109e78e>] kthread+0x9e/0xc0
<4> [<ffffffff8100c28a>] child_rip+0xa/0x20
<4> [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
<4> [<ffffffff8100c280>] ? child_rip+0x0/0x20
<4>Code: 89 6d f8 0f 1f 44 00 00 31 db 4c 8d af 88 00 00 00 49 89 fc 4c 89 ef e8 13 b6 a0 e0 49 8b 44 24 68 49 8d 54 24 68 48 39 d0 74 04 <48> 8b 58 10 4c 89 e8 66 ff 00 66 66 90 f6 05 26 4e c7 ff 08 74 
<1>RIP  [<ffffffffa0b2222c>] distribute_txn_get_next_transno+0x3c/0xd0 [ptlrpc]
<4> RSP <ffff88028674fca0>

Immediately before the crash the following errors are printed to lola-8's message file:

lola-8.log:Feb 24 00:01:06 lola-8 kernel: LustreError: 6612:0:(osp_object.c:588:osp_attr_get()) soaked-MDT0003-osp-MDT0002:osp_attr_get update error [0x200000009:0x3:0x0]: rc = -5
lola-8.log:Feb 24 00:01:06 lola-8 kernel: LustreError: 6612:0:(lod_sub_object.c:959:lod_sub_prep_llog()) soaked-MDT0002-mdtlov: can't get id from catalogs: rc = -5
lola-8.log:Feb 24 00:01:06 lola-8 kernel: LustreError: 6612:0:(lod_dev.c:419:lod_sub_recovery_thread()) soaked-MDT0003-osp-MDT0002 getting update log failed: rc = -5
...
...
lola-8.log:Feb 24 00:01:09 lola-8 kernel: LustreError: 6617:0:(update_records.c:72:update_records_dump()) master transno = 8594544408 batchid = 4299976565 flags = 0 ops = 73 params = 46
lola-8.log:Feb 24 00:01:09 lola-8 kernel: LustreError: 6617:0:(update_records.c:72:update_records_dump()) master transno = 8594544409 batchid = 4299976566 flags = 0 ops = 73 params = 46
lola-8.log:Feb 24 00:01:09 lola-8 kernel: LustreError: 6617:0:(update_records.c:72:update_records_dump()) master transno = 8594544411 batchid = 4299976567 flags = 0 ops = 73 params = 46
lola-8.log:Feb 24 00:01:09 lola-8 kernel: LustreError: 6617:0:(update_records.c:72:update_records_dump()) master transno = 8594544417 batchid = 4299976568 flags = 0 ops = 73 params = 46
lola-8.log:Feb 24 00:01:09 lola-8 kernel: general protection fault: 0000 [#1] 
lola-8.log:Feb 24 00:01:09 lola-8 kernel: LustreError: 6683:0:(ldlm_lib.c:2562:target_stop_recovery_thread()) soaked-MDT0002: Aborting recovery

Attached files: message, console, vmcore-dmesg.txt of lola-8.
Crash file is available, too.

Whamcloud Community JIRA

[LU-7809] general protection fault: 0000 during failback of MDS disk resources