[LU-3021] replay-vbr test 11a: RIP: ldiskfs_mb_release_inode_pa+0x346/0x360 [ldiskfs]

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.5.0
• Affects Version/s: Lustre 2.1.5

Environment

Lustre Tag: v2_1_5_RC1
Lustre Build: http://build.whamcloud.com/job/lustre-b2_1/191/
Distro/Arch: RHEL6.3/x86_64 (kernel version: 2.6.32_279.19.1.el6)
Network: IB (OFED 1.5.4)
ENABLE_QUOTA=yes

• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: Jian Yu

Description

While running replay-vbr test 11a, unmounting the MDS hung and the following errors occurred on MDS:

LDISKFS-fs error (device sdc5): ldiskfs_mb_release_inode_pa: pa free mismatch: [pa ffff8804165f3a58] [phy 77568] [logic 0] [len 2048] [free 2047] [error 0] [inode 13] [freed 2048]
Aborting journal on device sdc5-8.
Write to readonly device sdc (0x800025) bi_flags: f000000000000001, bi_vcnt: 1, bi_idx: 0, bi->size: 4096, bi_cnt: 2, bi_private: ffff880216d0b9b8
LDISKFS-fs (sdc5): Remounting filesystem read-only
Write to readonly device sdc (0x800025) bi_flags: f000000000000001, bi_vcnt: 1, bi_idx: 0, bi->size: 4096, bi_cnt: 2, bi_private: ffff88020ed844d8
LDISKFS-fs error (device sdc5): ldiskfs_mb_release_inode_pa: free 2048, pa_free 2047
------------[ cut here ]------------
kernel BUG at /var/lib/jenkins/workspace/lustre-b2_1/arch/x86_64/build_type/server/distro/el6/ib_stack/ofa/BUILD/BUILD/lustre-ldiskfs-3.3.0/ldiskfs/mballoc.c:3789!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:14.4/0000:01:04.0/local_cpus
CPU 0 
Modules linked in: cmm(U) osd_ldiskfs(U) mdt(U) mdd(U) mds(U) fsfilt_ldiskfs(U) mgs(U) mgc(U) ldiskfs(U) lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) jbd2 nfs fscache mlx4_ib(U) mlx4_core(U) nfsd lockd nfs_acl auth_rpcgss exportfs autofs4 sunrpc cpufreq_ondemand powernow_k8 freq_table mperf ib_ipoib(U) rdma_ucm(U) ib_ucm(U) ib_uverbs(U) ib_umad(U) rdma_cm(U) ib_cm(U) iw_cm(U) ib_addr(U) ipv6 ib_sa(U) ib_mad(U) ib_core(U) igb dca microcode serio_raw k10temp amd64_edac_mod edac_core edac_mce_amd i2c_piix4 i2c_core sg shpchp ext3 jbd mbcache sd_mod crc_t10dif pata_acpi ata_generic pata_atiixp ahci dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs]

Pid: 14217, comm: umount Not tainted 2.6.32-279.19.1.el6_lustre.x86_64 #1 Supermicro H8DGT/H8DGT
RIP: 0010:[<ffffffffa03c7ac6>]  [<ffffffffa03c7ac6>] ldiskfs_mb_release_inode_pa+0x346/0x360 [ldiskfs]
RSP: 0018:ffff88020c519a58  EFLAGS: 00010212
RAX: 00000000000007ff RBX: 0000000000000800 RCX: ffff880218f6c400
RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff8802151fc100
RBP: ffff88020c519b08 R08: ffffffff81c01a80 R09: 0000000000000000
R10: 0000000000000003 R11: 0000000000000000 R12: ffff8800ab859ef8
R13: ffff8800b84833a0 R14: 0000000000003801 R15: ffff8804165f3a58
FS:  00007fa7c0ea2740(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000003c8b873e10 CR3: 00000000a7379000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process umount (pid: 14217, threadinfo ffff88020c518000, task ffff880218df8080)
Stack:
 ffff880200000800 00000000000007ff ffff880000000000 000000000000000d
<d> 0000000000000800 0000000000000004 ffff88020c519a98 ffffffff811a8df6
<d> ffff880218f6c400 ffff88021dbe6800 ffff8804165f3a58 000000000000ff00
Call Trace:
 [<ffffffff811a8df6>] ? __wait_on_buffer+0x26/0x30
 [<ffffffffa03cb86e>] ldiskfs_discard_preallocations+0x1fe/0x490 [ldiskfs]
 [<ffffffffa03e3286>] ldiskfs_clear_inode+0x16/0x50 [ldiskfs]
 [<ffffffff81190c4c>] clear_inode+0xac/0x140
 [<ffffffff81190d20>] dispose_list+0x40/0x120
 [<ffffffff811911ca>] invalidate_inodes+0xea/0x190
 [<ffffffff8117859c>] generic_shutdown_super+0x4c/0xe0
 [<ffffffff81178661>] kill_block_super+0x31/0x50
 [<ffffffff81179670>] deactivate_super+0x70/0x90
 [<ffffffff811955df>] mntput_no_expire+0xbf/0x110
 [<ffffffffa0f912b4>] unlock_mntput+0x64/0x70 [obdclass]
 [<ffffffffa051b715>] ? cfs_waitq_init+0x15/0x20 [libcfs]
 [<ffffffffa0f993f3>] server_put_super+0x433/0x13e0 [obdclass]
 [<ffffffff811911d6>] ? invalidate_inodes+0xf6/0x190
 [<ffffffff811785ab>] generic_shutdown_super+0x5b/0xe0
 [<ffffffff81178696>] kill_anon_super+0x16/0x60
 [<ffffffffa0f8fa56>] lustre_kill_super+0x36/0x60 [obdclass]
 [<ffffffff81179670>] deactivate_super+0x70/0x90
 [<ffffffff811955df>] mntput_no_expire+0xbf/0x110
 [<ffffffff81195f3b>] sys_umount+0x7b/0x3a0
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Code: 55 c8 e9 39 fe ff ff 31 db 41 83 7f 4c 00 0f 84 7e fd ff ff 0f 0b eb fe 0f 0b eb fe 0f 0b 0f 1f 80 00 00 00 00 eb f7 0f 0b eb fe <0f> 0b 0f 1f 84 00 00 00 00 00 eb f6 66 66 66 66 66 2e 0f 1f 84 
RIP  [<ffffffffa03c7ac6>] ldiskfs_mb_release_inode_pa+0x346/0x360 [ldiskfs]
 RSP <ffff88020c519a58>

Maloo report: https://maloo.whamcloud.com/test_sets/29d0cb1e-943a-11e2-93c6-52540035b04c

Issue Links

• is duplicated by: LU-3330 mds crashed during umount after mds-survey run