[LU-7456] kernel panic on ldiskfs_journal_commit_callback+0x4a/0x80

• Type: Bug
• Resolution: Duplicate
• Priority: Critical
• Fix Version/s: Lustre 2.8.0
• Labels: dne2 soak
• Environment: lola; build 20151120
• Severity: 3
• Assignee: Di Wang
• Reporter: Di Wang

Description

<4>general protection fault: 0000 1 SMP
<4>last sysfs file: /sys/devices/system/cpu/online
<4>CPU 16
<4>Modules linked in: mgs(U) osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) ldiskfs(U) jbd2 8021q garp stp llc nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm scsi_dh_rdac dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core i2c_i801 ioatdma sg igb dca i2c_algo_bit i2c_core ptp pps_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas wmi mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
<4>
<4>Pid: 15263, comm: jbd2/dm-10-8 Tainted: P --------------- 2.6.32-504.30.3.el6_lustre.gf1f8275.x86_64 #1 Intel Corporation S2600GZ ........../S2600GZ
<4>RIP: 0010:[<ffffffffa082023a>] [<ffffffffa082023a>] ldiskfs_journal_commit_callback+0x4a/0x80 [ldiskfs]
<4>RSP: 0000:ffff88072c25fd00 EFLAGS: 00010287
<4>RAX: ffff880643ba1318 RBX: 0033332e312e7473 RCX: 0033332e312e7473
<4>RDX: ffff880643ba1318 RSI: ffff8806f0e6f958 RDI: ffff8807f32d5000
<4>-----------[ cut here ]-----------
<4>WARNING: at lib/list_debug.c:48 list_del+0x6e/0xa0() (Tainted: P --------------- )
<4>Hardware name: S2600GZ ..........
<4>list_del corruption. prev->next should be ffff8806438eb000, but was 6574646d2e726964
<4>Modules linked in: mgs(U) osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgc(U) osd_ldiskfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) ldiskfs(U) jbd2 8021q garp stp llc nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm scsi_dh_rdac dm_round_robin dm_multipath microcode iTCO_wdt iTCO_vendor_support zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) sb_edac edac_core lpc_ich mfd_core i2c_i801 ioatdma sg igb dca i2c_algo_bit i2c_core ptp pps_core ext3 jbd mbcache sd_mod crc_t10dif ahci isci libsas wmi mpt2sas scsi_transport_sas raid_class mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
<4>Pid: 7080, comm: osp_up4-2 Tainted: P --------------- 2.6.32-504.30.3.el6_lustre.gf1f8275.x86_64 #1
<4>Call Trace:
<4> [<ffffffff81074e47>] ? warn_slowpath_common+0x87/0xc0
<4> [<ffffffff81074f36>] ? warn_slowpath_fmt+0x46/0x50
<4> [<ffffffff81064c12>] ? default_wake_function+0x12/0x20
<4> [<ffffffff8129f63e>] ? list_del+0x6e/0xa0
<4> [<ffffffff81176008>] ? free_block+0xc8/0x170
<4> [<ffffffff81176139>] ? __drain_alien_cache+0x89/0xa0
<4> [<ffffffff8117658b>] ? kfree+0x26b/0x320
<4> [<ffffffffa0bdeed0>] ? ptlrpc_free_bulk+0x1d0/0x5a0 [ptlrpc]
<4> [<ffffffffa0bdf505>] ? __ptlrpc_req_finished+0x265/0x660 [ptlrpc]
<4> [<ffffffffa0bdf961>] ? ptlrpc_free_request+0x61/0x70 [ptlrpc]
<4> [<ffffffffa0bdfaa8>] ? ptlrpc_free_committed+0x138/0x770 [ptlrpc]
<4> [<ffffffffa0be1b91>] ? after_reply+0x7c1/0xe50 [ptlrpc]
<4> [<ffffffff8108742c>] ? lock_timer_base+0x3c/0x70
<4> [<ffffffffa0be5680>] ? ptlrpc_check_set+0x1270/0x1bd0 [ptlrpc]
<4> [<ffffffffa0be632a>] ? ptlrpc_set_wait+0x34a/0xa20 [ptlrpc]
<4> [<ffffffff81064c00>] ? default_wake_function+0x0/0x20
<4> [<ffffffffa0bf2565>] ? lustre_msg_set_jobid+0xf5/0x130 [ptlrpc]
<4> [<ffffffffa0be6a81>] ? ptlrpc_queue_wait+0x81/0x220 [ptlrpc]
<4> [<ffffffffa145ad16>] ? osp_send_update_req+0x256/0x850 [osp]
<4> [<ffffffffa145791c>] ? osp_get_next_request+0xec/0x1c0 [osp]
<4> [<ffffffffa145bca7>] ? osp_send_update_thread+0x557/0xb48 [osp]
<4> [<ffffffff81064c00>] ? default_wake_function+0x0/0x20
<4> [<ffffffffa145b750>] ? osp_send_update_thread+0x0/0xb48 [osp]
<4> [<ffffffff8109e78e>] ? kthread+0x9e/0xc0
<4> [<ffffffff8100c28a>] ? child_rip+0xa/0x20
<4> [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
<4> [<ffffffff8100c280>] ? child_rip+0x0/0x20
<4>--[ end trace 699bb73d7bb88148 ]--
<4>RBP: ffff88072c25fd20 R08: 0000000000000002 R09: 5a5a5a5a5a5a5a5a
<4>R10: 5a5a5a5a5a5a5a5a R11: 0000000000000000 R12: ffff8806f0e6f958
<4>R13: 0000000000000000 R14: ffff8807f32d5000 R15: 00000000000121c3
<4>FS: 0000000000000000(0000) GS:ffff880038300000(0000) knlGS:0000000000000000
<4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<4>CR2: 0000000001cf8888 CR3: 0000000001a85000 CR4: 00000000000407e0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process jbd2/dm-10-8 (pid: 15263, threadinfo ffff88072c25e000, task ffff8807f989f520)
<4>Stack:
<4> ffff880813b7539c ffff8806f0e6f880 ffff880813b75000 0000000000000000
<4><d> ffff88072c25fe60 ffffffffa07d574f ffff880038215900 ffffffff81a8d6c0
<4><d> 00ff8807f989f520 ffff8803cda8f828 ffff8806f0e6f880 ffff880813b7539c
<4>Call Trace:
<4> [<ffffffffa07d574f>] jbd2_journal_commit_transaction+0x10df/0x1500 [jbd2]
<4> [<ffffffff8108802b>] ? try_to_del_timer_sync+0x7b/0xe0
<4> [<ffffffffa07daa58>] kjournald2+0xb8/0x220 [jbd2]
<4> [<ffffffff8109ec20>] ? autoremove_wake_function+0x0/0x40
<4> [<ffffffffa07da9a0>] ? kjournald2+0x0/0x220 [jbd2]
<4> [<ffffffff8109e78e>] kthread+0x9e/0xc0
<4> [<ffffffff8100c28a>] child_rip+0xa/0x20
<4> [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
<4> [<ffffffff8100c280>] ? child_rip+0x0/0x20
<4>Code: 00 00 4c 8b b7 d0 04 00 00 41 83 e5 02 4c 39 e0 48 8b 18 48 89 c2 74 38 48 89 d9 eb 07 0f 1f 40 00 48 89 cb 48 8b 70 08 4c 89 f7 <48> 89 71 08 48 89 0e 48 89 c6 48 89 10 48 89 50 08 44 89 ea ff
<1>RIP [<ffffffffa082023a>] ldiskfs_journal_commit_callback+0x4a/0x80 [ldiskfs]

Issue Links

• is related to: LU-7455 Tracking tickets to make DNE pass soak-test.