[LU-7800] Panic during recovery of soak-test.

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.9.0
• Affects Version/s: Lustre 2.8.0
• Labels: llnl
• Severity: 3
• Assignee: Di Wang
• Reporter: Di Wang

Description

<3>LustreError: Skipped 3 previous similar messages
<4>Lustre: 4324:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1456026889/real 1456026889]  req@ffff88040bf736c0 x1526755024282608/t0(0) o38->soaked-MDT0005-osp-MDT0001@192.168.1.111@o2ib10:24/4 lens 520/544 e 0 to 1 dl 1456026900 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
<1>BUG: unable to handle kernel NULL pointer dereference at (null)
<1>IP: [<ffffffff8152cb13>] down_write+0x23/0x40
<4>PGD 0
<4>Oops: 0002 [#1] SMP
<4>last sysfs file: /sys/devices/system/cpu/online
<4>CPU 12
<4>Modules linked in: mgs(U) osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) ldiskfs(U) jbd2 lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) 8021q garp stp llc nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm scsi_dh_rdac dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core i2c_i801 ioatdma sg igb dca i2c_algo_bit i2c_core ptp pps_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas wmi mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
<4>
<4>Pid: 4408, comm: mdt03_002 Tainted: P           ---------------    2.6.32-504.30.3.el6_lustre.gf9ca359.x86_64 #1 Intel Corporation SandyBridge Platform/To be filled by O.E.M.
<4>RIP: 0010:[<ffffffff8152cb13>]  [<ffffffff8152cb13>] down_write+0x23/0x40
<4>RSP: 0018:ffff88081ba27810  EFLAGS: 00010246
<4>RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff880813088000
<4>RDX: ffffffff00000001 RSI: ffff88080753c0c0 RDI: 0000000000000000
<4>RBP: ffff88081ba27820 R08: ffff8808345c0e40 R09: 0000000000000000
<4>R10: ffff8808387c4fa0 R11: 0000000000000640 R12: ffff88080753c0c0
<4>R13: 0000000000000000 R14: ffff8808387c4f80 R15: ffff8808345c0e40
<4>FS:  0000000000000000(0000) GS:ffff88044e480000(0000) knlGS:0000000000000000
<4>CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<4>CR2: 0000000000000000 CR3: 0000000001a85000 CR4: 00000000000407e0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process mdt03_002 (pid: 4408, threadinfo ffff88081ba26000, task ffff88081b99a040)
<4>Stack:
<4> 0000000000000282 0000000000000000 ffff88081ba27880 ffffffffa0845ad3
<4><d> 0000000000000000 ffff8808387c4f80 ffff880813088000 ffff8808334e26c0
<4><d> ffff88081bbe83c0 ffff88080753c0c0 ffff8808334e26c0 ffff880813088000
<4>Call Trace:
<4> [<ffffffffa0845ad3>] llog_cat_add_rec+0x403/0x7b0 [obdclass]
<4> [<ffffffffa083c239>] llog_add+0x89/0x1c0 [obdclass]
<4> [<ffffffffa0b1eea4>] sub_updates_write+0x1b4/0x12b0 [ptlrpc]
<4> [<ffffffffa0b2080c>] top_trans_stop+0x86c/0xbd0 [ptlrpc]
<4> [<ffffffffa12e160c>] ? mdd_links_write+0xac/0x210 [mdd]
<4> [<ffffffffa08a5c10>] ? lu_ucred+0x20/0x30 [obdclass]
<4> [<ffffffffa125981c>] lod_trans_stop+0x2bc/0x330 [lod]
<4> [<ffffffffa126f608>] ? lod_object_write_unlock+0x38/0xd0 [lod]
<4> [<ffffffffa13002fa>] mdd_trans_stop+0x1a/0xac [mdd]
<4> [<ffffffffa12ea8d8>] mdd_create+0x1368/0x1770 [mdd]
<4> [<ffffffffa11a40f2>] ? mdt_version_check+0x132/0x440 [mdt]
<4> [<ffffffffa11add5c>] mdt_reint_create+0xbdc/0xfe0 [mdt]
<4> [<ffffffffa11a318d>] mdt_reint_rec+0x5d/0x200 [mdt]
<4> [<ffffffffa118eddb>] mdt_reint_internal+0x62b/0x9f0 [mdt]
<4> [<ffffffffa118f63b>] mdt_reint+0x6b/0x120 [mdt]
<4> [<ffffffffa0b0bc3c>] tgt_request_handle+0x8ec/0x1440 [ptlrpc]
<4> [<ffffffffa0ab8c61>] ptlrpc_main+0xd21/0x1800 [ptlrpc]
<4> [<ffffffff8152a39e>] ? thread_return+0x4e/0x7d0
<4> [<ffffffffa0ab7f40>] ? ptlrpc_main+0x0/0x1800 [ptlrpc]
<4> [<ffffffff8109e78e>] kthread+0x9e/0xc0
<4> [<ffffffff8100c28a>] child_rip+0xa/0x20

Issue Links

• is related to: LU-7844 Rolling upgrade: sanity test_61 FAIL: BUG: unable to handle kernel NULL pointer dereference at (null)
• is related to: LU-8484 MDS server crash during sanity test 63a run
• is related to: LU-8370 ASSERTION( lur->lur_hdr.lrh_len <= ctxt->loc_chunk_size )
• is related to: LU-8489 llog_cat_add_rec NULL pointer dereference