[LU-5336] kernel BUG at fs/jbd2/transaction.c:1033! (dup of LU-3102?)

• Type: Bug
• Resolution: Duplicate
• Priority: Critical
• Affects Version/s: Lustre 2.4.3
• Severity: 4
• Assignee: Niu Yawei (Inactive)
• Reporter: Mahmoud Hanafi

Description

After disabling/enabling quotas (tune2fs -O ^quota;tune2fs -O quota) 3 OSS start to crash over and over with this bug.
We where able to bring up the filesystem only after disabling quota on all the OSTs on the 3 OSS servers.

We have LU-4382 applied.

Here is the stack trace

------------[ cut here ]------------^M
kernel BUG at fs/jbd2/transaction.c:1033!^M
^M
Entering kdb (current=0xffff881fb31ceaa0, pid 12018) on processor 12 due to KDB_ENTER()^M
[12]kdb> [-- root@localhost attached -- Sun Jul 13 06:44:41 2014]^M
^M
[12]kdb> sr 9^H ^M[12]kdb> sr 8^M
SysRq : Changing Loglevel^M
Loglevel set to 8^M
[12]kdb> sr -^H ^M[12]kdb> sr p^M
SysRq : Show Regs^M
CPU 12 ^M
Modules linked in: ofd(U) ost(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) mdd(U) ldiskfs(U) jbd2 acpi_cpufreq freq_table mperf lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) dm_round_robin scsi_dh_rdac lpfc(U) scsi_transport_fc scsi_tgt nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc bonding 8021q garp stp llc ib_ucm(U) rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) mlx4_ib(U) ib_mad(U) ib_core(U) dm_multipath tcp_bic microcode sg mlx4_core(U) compat(U) igb dca i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ext3 jbd sd_mod crc_t10dif mpt2sas raid_class isci libsas scsi_transport_sas ahci wmi dm_mirror dm_region_hash dm_log dm_mod gru [last unloaded: scsi_wait_scan]^M
^M
Pid: 12018, comm: ll_ost03_054 Not tainted 2.6.32-358.23.2.el6.20140115.x86_64.lustre243 #1 SGI.COM SUMMIT/S2600GZ^M
RIP: 0010:[<ffffffffa0bd68ad>]  [<ffffffffa0bd68ad>] jbd2_journal_dirty_metadata+0x10d/0x150 [jbd2]^M
RSP: 0018:ffff881fb119d960  EFLAGS: 00010246^M
RAX: ffff881faea87a80 RBX: ffff881ebc31b738 RCX: ffff880fcb8d2950^M
RDX: 0000000000000000 RSI: ffff880fcb8d2950 RDI: 0000000000000000^M
RBP: ffff881fb119d980 R08: ffff880fcb8d2950 R09: f01676c993425402^M
R10: ffff880ff65dcc00 R11: 0000000000000000 R12: ffff882003f15c88^M
R13: ffff880fcb8d2950 R14: ffff880f66694000 R15: 0000000000000000^M
FS:  00007fffedaf3700(0000) GS:ffff881078880000(0000) knlGS:0000000000000000^M
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b^M
CR2: 0000003f62aacff0 CR3: 0000000001a25000 CR4: 00000000000407e0^M
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000^M
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400^M
Process ll_ost03_054 (pid: 12018, threadinfo ffff881fb119c000, task ffff881fb31ceaa0)^M
Stack:^M
 ffff881ebc31b738 ffffffffa0c3a030 ffff880fcb8d2950 0000000000000000^M
<d> ffff881fb119d9c0 ffffffffa0bfa0bb ffffffffa0c3a010 ffff881ebc31b738^M
<d> ffff880e8fc75600 ffff880feea752d0 ffff880feea75200 ffff881fb119da40^M
Call Trace:^M
 [<ffffffffa0bfa0bb>] __ldiskfs_handle_dirty_metadata+0x7b/0x100 [ldiskfs]^M
 [<ffffffffa0c059ba>] ldiskfs_mark_iloc_dirty+0x52a/0x630 [ldiskfs]^M
 [<ffffffffa0c06fc3>] ldiskfs_mark_inode_dirty+0x83/0x1f0 [ldiskfs]^M
 [<ffffffffa0c08c00>] ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]^M
 [<ffffffffa0d62e31>] osd_attr_set+0x181/0x540 [osd_ldiskfs]^M
 [<ffffffffa0e32879>] dt_attr_set.clone.2+0x29/0xc0 [ofd]^M
 [<ffffffffa0e36362>] ofd_attr_set+0x522/0x6c0 [ofd]^M
 [<ffffffffa0e27e2a>] ofd_setattr+0x69a/0xb80 [ofd]^M
 [<ffffffffa0df8c1c>] ost_setattr+0x31c/0x990 [ost]^M
 [<ffffffffa0dfc746>] ost_handle+0x21e6/0x48e0 [ost]^M
 [<ffffffffa04940f4>] ? libcfs_id2str+0x74/0xb0 [libcfs]^M
 [<ffffffffa077e3b8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]^M
 [<ffffffffa04885de>] ? cfs_timer_arm+0xe/0x10 [libcfs]^M
 [<ffffffffa0499d3f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]^M
 [<ffffffffa0775719>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]^M
 [<ffffffff81055813>] ? __wake_up+0x53/0x70^M
 [<ffffffffa077f74e>] ptlrpc_main+0xace/0x1700 [ptlrpc]^M
 [<ffffffffa077ec80>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]^M
 [<ffffffff8100c0ca>] child_rip+0xa/0x20^M
Code: c6 9c 03 00 00 4c 89 f7 e8 11 b7 96 e0 48 8b 33 ba 01 00 00 00 4c 89 e7 e8 11 ec ff ff 4c 89 f0 66 ff 00 66 66 90 e9 73 ff ff ff <0f> 0b eb fe 0f 0b eb fe 0f 0b 66 0f 1f 84 00 00 00 00 00 eb f5 ^M
Call Trace:^M
 [<ffffffffa0bfa0bb>] __ldiskfs_handle_dirty_metadata+0x7b/0x100 [ldiskfs]^M
 [<ffffffffa0c059ba>] ldiskfs_mark_iloc_dirty+0x52a/0x630 [ldiskfs]^M
 [<ffffffffa0c06fc3>] ldiskfs_mark_inode_dirty+0x83/0x1f0 [ldiskfs]^M
 [<ffffffffa0c08c00>] ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]^M
 [<ffffffffa0d62e31>] osd_attr_set+0x181/0x540 [osd_ldiskfs]^M
 [<ffffffffa0e32879>] dt_attr_set.clone.2+0x29/0xc0 [ofd]^M
                                                                                                            29999,1       85%
 [<ffffffffa0d62e31>] osd_attr_set+0x181/0x540 [osd_ldiskfs]^M
 [<ffffffffa0e32879>] dt_attr_set.clone.2+0x29/0xc0 [ofd]^M
 [<ffffffffa0e36362>] ofd_attr_set+0x522/0x6c0 [ofd]^M
 [<ffffffffa0e27e2a>] ofd_setattr+0x69a/0xb80 [ofd]^M
 [<ffffffffa0df8c1c>] ost_setattr+0x31c/0x990 [ost]^M
 [<ffffffffa0dfc746>] ost_handle+0x21e6/0x48e0 [ost]^M
 [<ffffffffa04940f4>] ? libcfs_id2str+0x74/0xb0 [libcfs]^M
 [<ffffffffa077e3b8>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]^M
 [<ffffffffa04885de>] ? cfs_timer_arm+0xe/0x10 [libcfs]^M
 [<ffffffffa0499d3f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]^M
 [<ffffffffa0775719>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]^M
 [<ffffffff81055813>] ? __wake_up+0x53/0x70^M
 [<ffffffffa077f74e>] ptlrpc_main+0xace/0x1700 [ptlrpc]^M
 [<ffffffffa077ec80>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]^M
 [<ffffffff8100c0ca>] child_rip+0xa/0x20^M
 [<ffffffffa077ec80>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]^M
 [<ffffffffa077ec80>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]^M
 [<ffffffff8100c0c0>] ? child_rip+0x0/0x20^M

Issue Links

• is related to: LU-5040 kernel BUG at fs/jbd2/transaction.c:1033