https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1948 Lustre <p>Sep 16 01:28:12 hyperion-rst6 kernel: LDISKFS-fs error (device md1): ldiskfs_add_entry:<br/> Sep 16 01:28:12 hyperion-rst6 kernel: LDISKFS-fs error (device md1): ldiskfs_add_entry: bad entry in directory #88606751: rec_len is smaller than minimal - block=44369457offset=536(536), inode=88627711, rec_len=0, name_len=4<br/> Sep 16 01:28:12 hyperion-rst6 kernel: Aborting journal on device md1-8.<br/> Sep 16 01:28:12 hyperion-rst6 kernel: LDISKFS-fs error (device md1) in ldiskfs_reserve_inode_write: Journal has aborted<br/> Sep 16 01:28:12 hyperion-rst6 kernel: LDISKFS-fs (md1): Remounting filesystem read-only<br/> Sep 16 01:28:12 hyperion-rst6 kernel: LDISKFS-fs (md1): Remounting filesystem read-only<br/> Sep 16 01:28:12 hyperion-rst6 kernel: LDISKFS-fs error (device md1) in ldiskfs_new_inode: Journal has aborted<br/> Sep 16 01:28:12 hyperion-rst6 kernel: LDISKFS-fs error (device md1) in ldiskfs_delete_inode: Journal has aborted<br/> Sep 16 01:28:12 hyperion-rst6 kernel: LustreError: 4489:0:(osd_io.c:1014:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30<br/> Sep 16 01:28:12 hyperion-rst6 kernel: LustreError: 4186:0:(osd_handler.c:894:osd_trans_stop()) Failure in transaction hook: -30</p> <p>Disk appeared to be quite messed up with fsck -fy. Ran the data capture script from lu-1015, results attached.</p> SWL - Hyperion/LLNL RHEL6 servers and clients LU-1948

ldiskfs - MDS goes read-only (SWL)

Bug Blocker Resolved Duplicate Di Wang Cliff White Sun, 16 Sep 2012 11:41:43 +0000 Tue, 9 Oct 2012 00:55:11 +0000 Mon, 1 Oct 2012 17:20:18 +0000 Lustre 2.3.0 0 8 <p>Bobijam</p> <p>Is there anything that you can determine from this report?</p> <p>Peter</p> <p>Cliff, Could you tell me more about your test? on lu-1015, I saw most discussion is about OST fails and run fsck. <br/> Did you do the same thing about MDS? run tests, uncleanup power off, fsck -p (found problem), then run fsck -fy, start MDS again, found these problem? What test did you run? Could you please post the script here? And I assume you use e2fsprogs-1.42.3.wc3? </p> <p>The test being run is SWL, which is essentially a large mix of multiple tasks (ior,fdtree,mib,etc) In this case, the MDS went read-only in the<br/> middle of the test. I halted the MDS and ran the data-dump script created for <a href="https://jira.whamcloud.com/browse/LU-1015" title="ldiskfs corruption with large LUNs" class="issue-link" data-issue-key="LU-1015"><del>LU-1015</del></a>. e2fsprogs was 1.41.90-wc4-7, i will reinstall newer.</p> <p>current script:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #!/bin/bash ostdev=$1 logdir=/home/white215/wham/lu1015/logs/ stamp=`date +%Y%m%d.%M%S` dumpe2fs /dev/$ostdev &gt; $logdir/$ostdev.$stamp.stats debugfs -c -R <span class="code-quote">"dump &lt;8&gt;"</span> /dev/$ostdev &gt; $logdir/$ostdev.$stamp.journal debugfs -c -R <span class="code-quote">"logdump -a"</span> /dev/$ostdev &gt; $logdir/$ostdev.$stamp.logdump e2fsck -fp /dev/$ostdev 2&gt;&amp;1 | tee $logdir/$ostdev.$stamp.e2fsck dumpe2fs /dev/$ostdev &gt; $logdir/$ostdev.$stamp.stats.post </pre> </div></div> <p>Thanks, I see. Is the test started from a fresh reformatted MDS? or the MDS was just been fixed by e2fsck -fy before the run? If the MDS has not been changed yet, could you find out the directory name of 88606751? I guess you can just mount it as ldiskfs, and get it by find /mnt/mds -inum 88606751 Thanks.</p> <p>MDS just went read-only, then crashed when I attempted a umount -f</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2012-09-17 17:44:00 Lustre: 4130:0:(service.c:2101:ptlrpc_handle_rs()) All locks stolen from rs ffff880132a5d000 x1413385599017471.t4342523245 o0 NID 192.168.119.37@o2ib1 2012-09-17 17:44:00 Lustre: 4130:0:(service.c:2101:ptlrpc_handle_rs()) Skipped 402 previous similar messages 2012-09-17 17:44:00 LustreError: 5729:0:(mdt_recovery.c:611:mdt_steal_ack_locks()) Skipped 426 previous similar messages 2012-09-17 17:47:49 LDISKFS-fs error (device md1): ldiskfs_add_entry: 2012-09-17 17:47:49 LDISKFS-fs error (device md1): ldiskfs_add_entry: bad entry in directory #116917774: rec_len is smaller than minimal - block=58525219offset=312(312), inode=117441145, rec_len=0, name_len=4 2012-09-17 17:47:49 Aborting journal on device md1-8. 2012-09-17 17:47:49 LDISKFS-fs error (device md1) in ldiskfs_reserve_inode_write: Journal has aborted 2012-09-17 17:47:49 LDISKFS-fs error (device md1) in ldiskfs_reserve_inode_write: Journal has aborted 2012-09-17 17:47:49 LDISKFS-fs error (device md1) in ldiskfs_reserve_inode_write: Journal has aborted 2012-09-17 17:47:49 LDISKFS-fs error (device md1) in ldiskfs_reserve_inode_write: Journal has aborted 2012-09-17 17:47:49 LDISKFS-fs (md1): 2012-09-17 17:47:49 LustreError: 5731:0:(osd_io.c:1014:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30 2012-09-17 17:47:49 LDISKFS-fs (md1): 2012-09-17 17:47:49 LDISKFS-fs (md1): 2012-09-17 17:47:49 LDISKFS-fs (md1): Remounting filesystem read-onlyRemounting filesystem read-onlyRemounting filesystem read-only </pre> </div></div> <p>Kernel BUG</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2012-09-17 17:49:57 LDISKFS-fs error (device md1): ldiskfs_mb_release_inode_pa: pa free mismatch: [pa ffff88031438e748] [phy 190752] [logic 608] [len 16] [free 13] [error 0] [inode 7340126] [freed 16] 2012-09-17 17:49:57 LDISKFS-fs error (device md1): ldiskfs_mb_release_inode_pa: free 16, pa_free 13 2012-09-17 17:49:57 ------------[ cut here ]------------ 2012-09-17 17:49:57 kernel BUG at /<span class="code-keyword">var</span>/lib/jenkins/workspace/lustre-b2_3/arch/x86_64/build_type/server/distro/el6/ib_stack/inkernel/BUILD/BUILD/lustre-ldiskfs-3.3.0/ldiskfs/mballoc.c:3784! 2012-09-17 17:49:57 invalid opcode: 0000 [#1] SMP 2012-09-17 17:49:57 last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map 2012-09-17 17:49:57 CPU 14 2012-09-17 17:49:57 Modules linked in: cmm(U) osd_ldiskfs(U) mdt(U) mdd(U) mds(U) fsfilt_ldiskfs(U) exportfs mgs(U) mgc(U) lustre(U) lquota(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ldiskfs(U) mbcache jbd2 zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa mlx4_ib ib_mad ib_core dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun kvm raid0 sg sr_mod cdrom sd_mod crc_t10dif dcdbas serio_raw ata_generic pata_acpi ata_piix iTCO_wdt iTCO_vendor_support mptsas mptscsih mptbase scsi_transport_sas i7core_edac edac_core ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc mlx4_en mlx4_core bnx2 [last unloaded: scsi_wait_scan] 2012-09-17 17:49:57 2012-09-17 17:49:57 Pid: 10207, comm: umount Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.x86_64 #1 Dell Inc. PowerEdge R610/0K399H 2012-09-17 17:49:57 RIP: 0010:[&lt;ffffffffa07214b6&gt;] [&lt;ffffffffa07214b6&gt;] ldiskfs_mb_release_inode_pa+0x346/0x360 [ldiskfs] 2012-09-17 17:49:57 RSP: 0018:ffff88026e7152b8 EFLAGS: 00010216 2012-09-17 17:49:57 RAX: 000000000000000d RBX: 0000000000000010 RCX: ffff88032a805800 2012-09-17 17:49:57 RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff880316f3a2c0 2012-09-17 17:49:57 RBP: ffff88026e715368 R08: 0000000000000000 R09: 0000000000000000 2012-09-17 17:49:57 R10: 00000000001 R11: 0000000000000000 R12: ffff88025db65898 2012-09-17 17:49:57 R13: ffff88023f45d3a0 R14: 0000000000002931 R15: ffff88031438e748 2012-09-17 17:49:57 FS: 00002aaaab690740(0000) GS:ffff8801b58e0000(0000) knlGS:0000000000000000 2012-09-17 17:49:57 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b 2012-09-17 17:49:57 CR2: 00002aaaaace3387 CR3: 00000002b6ee5000 CR4: 00000000000006e0 2012-09-17 17:49:57 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2012-09-17 17:49:57 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2012-09-17 17:49:57 <span class="code-object">Process</span> umount (pid: 10207, threadinfo ffff88026e714000, task ffff88007fa10080) 2012-09-17 17:49:57 Stack: 2012-09-17 17:49:57 ffff880200000010 000000000000000d ffff880200000000 000000000070005e 2012-09-17 17:49:57 &lt;d&gt; 0000000000000010 000000000000000b ffff88026e7152f8 ffffffff811ae146 2012-09-17 17:49:57 &lt;d&gt; ffff88032a805800 ffff88032a3b8c00 ffff88031438e748 000000000002c000 2012-09-17 17:49:57 Call Trace: 2012-09-17 17:49:57 [&lt;ffffffff811ae146&gt;] ? __wait_on_buffer+0x26/0x30 2012-09-17 17:49:57 [&lt;ffffffffa072701e&gt;] ldiskfs_discard_preallocations+0x1fe/0x490 [ldiskfs] 2012-09-17 17:49:57 [&lt;ffffffffa0711077&gt;] ldiskfs_release_file+0xb7/0xd0 [ldiskfs] 2012-09-17 17:49:57 [&lt;ffffffff8117ca65&gt;] __fput+0xf5/0x210 2012-09-17 17:49:57 [&lt;ffffffff8117cba5&gt;] fput+0x25/0x30 2012-09-17 17:49:57 [&lt;ffffffff811785cd&gt;] filp_close+0x5d/0x90 2012-09-17 17:49:57 [&lt;ffffffffa07a1dbe&gt;] llog_lvfs_close+0x2e/0x130 [obdclass] 2012-09-17 17:49:57 [&lt;ffffffffa079afff&gt;] llog_close+0x5f/0x1b0 [obdclass] 2012-09-17 17:49:57 [&lt;ffffffffa079e92d&gt;] llog_cat_put+0x5d/0x160 [obdclass] 2012-09-17 17:49:57 [&lt;ffffffffa07a76bc&gt;] llog_obd_origin_cleanup+0x1bc/0x460 [obdclass] 2012-09-17 17:49:57 [&lt;ffffffffa07a7366&gt;] __llog_ctxt_put+0xc6/0x260 [obdclass] 2012-09-17 17:49:57 [&lt;ffffffffa07a7a0c&gt;] llog_cleanup+0xac/0x490 [obdclass] 2012-09-17 17:49:57 [&lt;ffffffffa0b7b6ef&gt;] osc_llog_finish+0x7f/0x250 [osc] 2012-09-17 17:49:57 [&lt;ffffffffa07a6438&gt;] obd_llog_finish+0x88/0x1a0 [obdclass] 2012-09-17 17:49:57 [&lt;ffffffffa0b76dca&gt;] osc_precleanup+0x2ea/0x3b0 [osc] 2012-09-17 17:49:57 [&lt;ffffffffa07d00c7&gt;] class_cleanup+0x1f7/0xdc0 [obdclass] 2012-09-17 17:49:58 [&lt;ffffffffa07b12b6&gt;] ? class_name2dev+0x56/0xe0 [obdclass] 2012-09-17 17:49:58 [&lt;ffffffffa07d243b&gt;] class_process_config+0x102b/0x1c30 [obdclass] 2012-09-17 17:49:58 [&lt;ffffffffa0398be0&gt;] ? cfs_alloc+0x30/0x60 [libcfs] 2012-09-17 17:49:58 [&lt;ffffffffa07cbe43&gt;] ? lustre_cfg_new+0x353/0x7e0 [obdclass] 2012-09-17 17:49:58 [&lt;ffffffffa07d31b9&gt;] class_manual_cleanup+0x179/0x6f0 [obdclass] 2012-09-17 17:49:58 [&lt;ffffffffa0c09743&gt;] lov_putref+0x393/0xb30 [lov] 2012-09-17 17:49:58 [&lt;ffffffff814ff18e&gt;] ? mutex_lock+0x1e/0x50 2012-09-17 17:49:58 [&lt;ffffffffa0c1335f&gt;] lov_disconnect+0x19f/0x4c0 [lov] 2012-09-17 17:49:58 [&lt;ffffffffa0eb1c72&gt;] mds_lov_clean+0x262/0x6f0 [mds] 2012-09-17 17:49:58 [&lt;ffffffffa0eb22c6&gt;] mds_precleanup+0x1c6/0x4b0 [mds] 2012-09-17 17:49:58 [&lt;ffffffffa07d00c7&gt;] class_cleanup+0x1f7/0xdc0 [obdclass] 2012-09-17 17:49:58 [&lt;ffffffff81271fe5&gt;] ? _atomic_dec_and_lock+0x55/0x80 2012-09-17 17:49:58 [&lt;ffffffffa07e9827&gt;] ? lu_object_put+0x157/0x290 [obdclass] 2012-09-17 17:49:58 [&lt;ffffffffa0eee8c4&gt;] mdd_fini_obd+0x54/0x240 [mdd] 2012-09-17 17:49:58 [&lt;ffffffffa0f0d777&gt;] mdd_process_config+0x2d7/0xa20 [mdd] 2012-09-17 17:49:58 [&lt;ffffffff814fdca0&gt;] ? thread_return+0x4e/0x76e 2012-09-17 17:49:58 [&lt;ffffffffa06a3b6b&gt;] cmm_process_config+0x7b/0xd10 [cmm] 2012-09-17 17:49:58 [&lt;ffffffffa0f6fd8a&gt;] mdt_stack_fini+0x5da/0xd40 [mdt] 2012-09-17 17:49:58 [&lt;ffffffffa06a2886&gt;] ? cmm_init_capa_ctxt+0x46/0x140 [cmm] 2012-09-17 17:49:58 [&lt;ffffffffa0f707c5&gt;] mdt_device_fini+0x2d5/0x5f0 [mdt] 2012-09-17 17:49:58 [&lt;ffffffffa07d0447&gt;] class_cleanup+0x577/0xdc0 [obdclass] 2012-09-17 17:49:58 [&lt;ffffffffa07b12b6&gt;] ? class_name2dev+0x56/0xe0 [obdclass] 2012-09-17 17:49:58 [&lt;ffffffffa07d243b&gt;] class_process_config+0x102b/0x1c30 [obdclass] 2012-09-17 17:49:58 [&lt;ffffffffa0398be0&gt;] ? cfs_alloc+0x30/0x60 [libcfs] 2012-09-17 17:49:58 [&lt;ffffffffa07cbe43&gt;] ? lustre_cfg_new+0x353/0x7e0 [obdclass] 2012-09-17 17:49:58 [&lt;ffffffffa07d31b9&gt;] class_manual_cleanup+0x179/0x6f0 [obdclass] 2012-09-17 17:49:58 [&lt;ffffffffa07b12b6&gt;] ? class_name2dev+0x56/0xe0 [obdclass] 2012-09-17 17:49:58 [&lt;ffffffffa07dd3c9&gt;] server_put_super+0x6f9/0xcf0 [obdclass] 2012-09-17 17:49:58 [&lt;ffffffff8117d34b&gt;] generic_shutdown_super+0x5b/0xe0 2012-09-17 17:49:58 [&lt;ffffffff8117d436&gt;] kill_anon_super+0x16/0x60 2012-09-17 17:49:58 [&lt;ffffffffa07d4df6&gt;] lustre_kill_super+0x36/0x60 [obdclass] 2012-09-17 17:49:58 [&lt;ffffffff8117e4b0&gt;] deactivate_super+0x70/0x90 2012-09-17 17:49:58 [&lt;ffffffff8119a4ff&gt;] mntput_no_expire+0xbf/0x110 2012-09-17 17:49:58 [&lt;ffffffff8119af9b&gt;] sys_umount+0x7b/0x3a0 2012-09-17 17:49:58 [&lt;ffffffff81082de1&gt;] ? sigprocmask+0x71/0x110 2012-09-17 17:49:58 [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b 2012-09-17 17:49:58 Code: 55 c8 e9 39 fe ff ff 31 db 41 83 7f 4c 00 0f 84 7e fd ff ff 0f 0b eb fe 0f 0b eb fe 0f 0b 0f 1f 80 00 00 00 00 eb f7 0f 0b eb fe &lt;0f&gt; 0b 0f 1f 84 00 00 00 00 00 eb f6 66 66 66 66 66 2e 0f 1f 84 2012-09-17 17:49:58 RIP [&lt;ffffffffa07214b6&gt;] ldiskfs_mb_release_inode_pa+0x346/0x360 [ldiskfs] 2012-09-17 17:49:58 RSP &lt;ffff88026e7152b8&gt; 2012-09-17 17:49:58 Initializing cgroup subsys cpuset </pre> </div></div> <p>am rebooting the node now, will run fsck script.</p> <p>I add some debug patch based on b2_3 (<a href="http://review.whamcloud.com/#change,4020" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4020</a> only patch on el6 ldiskfs series). build <a href="http://build.whamcloud.com/job/lustre-reviews/9250/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/9250/</a> Cliff, could you please test with this rpm. And I suspect this problem might be related with ldiskfs htree, so Liang is looking at this problem as well.</p> <p>Okay, the MDS just 'sploded. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2012-09-18 14:13:37 LDISKFS-fs error (device md1): add_dirent: 2012-09-18 14:13:37 LDISKFS-fs error (device md1): ldiskfs_add_entry: bad entry in directory #78644697: rec_len is smaller than minimal - block=39336951offset=152(152), inode=78655820, rec_len=0, name_len=4 2012-09-18 14:13:37 Aborting journal on device md1-8. 2012-09-18 14:13:37 LDISKFS-fs error (device md1): ldiskfs_journal_start_sb: Detected aborted journal 2012-09-18 14:13:37 LDISKFS-fs (md1): Remounting filesystem read-only 2012-09-18 14:13:37 LDISKFS-fs (md1): Remounting filesystem read-only 2012-09-18 14:13:37 LDISKFS-fs error (device md1): add_dirent: 78644697: lck ffff8801304ad000 off=152(78655820),inode=32, rec_len=4, name_len=0 2012-09-18 14:13:37 2012-09-18 14:13:37 LustreError: 8052:0:(osd_io.c:1014:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30 2012-09-18 14:13:37 LustreError: 8052:0:(osd_handler.c:894:osd_trans_stop()) Failure in transaction hook: -30 2012-09-18 14:13:37 LustreError: 8052:0:(osd_handler.c:899:osd_trans_stop()) Failure to stop transaction: -30 2012-09-18 14:13:37 78644697: name L0F7, lck 0000000000000004, name_len=568931904 2012-09-18 14:13:37 2012-09-18 14:13:37 ------------[ cut here ]------------ 2012-09-18 14:13:37 kernel BUG at /<span class="code-keyword">var</span>/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el6/ib_stack/inkernel/BUILD/BUILD/lustre-ldiskfs-3.3.0/ldiskfs/namei.c:1757! 2012-09-18 14:13:37 invalid opcode: 0000 [#1] SMP 2012-09-18 14:13:37 last sysfs file: /sys/devices/pci0000:00/0000:00:09.0/0000:05:00.0/infiniband_mad/umad0/port 2012-09-18 14:13:37 CPU 7 2012-09-18 14:13:37 Modules linked in: cmm(U) osd_ldiskfs(U) mdt(U) mdd(U) mds(U) fsfilt_ldiskfs(U) exportfs mgs(U) mgc(U) ldiskfs(U) mbcache jbd2 lustre(U) lquota(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa mlx4_ib ib_mad ib_core dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun kvm raid0 sg sr_mod cdrom sd_mod crc_t10dif dcdbas serio_raw ata_generic pata_acpi ata_piix iTCO_wdt iTCO_vendor_support mptsas mptscsih mptbase scsi_transport_sas i7core_edac edac_core ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc mlx4_en mlx4_core bnx2 [last unloaded: scsi_wait_scan] 2012-09-18 14:13:37 2012-09-18 14:13:37 Pid: 4898, comm: mdt03_008 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.gb4cc145.x86_64 #1 Dell Inc. PowerEdge R610/0K399H 2012-09-18 14:13:37 RIP: 0010:[&lt;ffffffffa0db21d0&gt;] [&lt;ffffffffa0db21d0&gt;] add_dirent_to_buf+0x4c0/0x530 [ldiskfs] 2012-09-18 14:13:37 RSP: 0018:ffff880121e935f0 EFLAGS: 00010246 2012-09-18 14:13:37 RAX: ffff880181248000 RBX: ffff880268654078 RCX: 00000000000014b5 2012-09-18 14:13:37 RDX: ffff880268654098 RSI: 0000000000000046 RDI: ffff88016ff93c00 2012-09-18 14:13:37 RBP: ffff880121e936b0 R08: 0000000000000000 R09: 0000000000000000 2012-09-18 14:13:37 R10: 0000000000000001 R11: 0000000000000000 R12: ffff880130e16ad0 2012-09-18 14:13:37 R13: 0000000000000000 R14: 0000000000000004 R15: 0000000000000020 2012-09-18 14:13:37 FS: 00002aaaab47e700(0000) GS:ffff880028260000(0000) knlGS:0000000000000000 2012-09-18 14:13:37 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b 2012-09-18 14:13:37 CR2: 00000000006d3a80 CR3: 0000000001a85000 CR4: 00000000000006e0 2012-09-18 14:13:37 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2012-09-18 14:13:37 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2012-09-18 14:13:37 <span class="code-object">Process</span> mdt03_008 (pid: 4898, threadinfo ffff880121e92000, task ffff88012df62080) 2012-09-18 14:13:37 LustreError: 4099:0:(fsfilt-ldiskfs.c:332:fsfilt_ldiskfs_start()) error starting handle <span class="code-keyword">for</span> op 10 (114 credits): rc -30 2012-09-18 14:13:37 LustreError: 4099:0:(llog_server.c:414:llog_origin_handle_cancel()) fsfilt_start_log() failed: -30 2012-09-18 14:13:37 LustreError: 4099:0:(llog_server.c:453:llog_origin_handle_cancel()) Cancel 0 of 121 llog-records failed: -30 2012-09-18 14:13:37 Stack: 2012-09-18 14:13:37 ffff880121e93640 0000000002583bf7 0000000000001000 0000000000000000 2012-09-18 14:13:37 &lt;d&gt; ffff8801abdad5c0 ffff880200000004 ffff880104b03141 0000000000000004 2012-09-18 14:13:37 &lt;d&gt; 00000020e2f30800 ffff8801258d9e00 ffff8801064c3288 ffff880268654fe0 2012-09-18 14:13:37 Call Trace: 2012-09-18 14:13:37 [&lt;ffffffffa0db5c5d&gt;] ldiskfs_add_entry+0xcd/0x500 [ldiskfs] 2012-09-18 14:13:37 [&lt;ffffffffa1001689&gt;] __osd_ea_add_rec+0xb9/0x190 [osd_ldiskfs] 2012-09-18 14:13:37 [&lt;ffffffffa100dbeb&gt;] osd_index_ea_insert+0x21b/0x5e0 [osd_ldiskfs] 2012-09-18 14:13:37 [&lt;ffffffffa0eee977&gt;] __mdd_index_insert_only+0x147/0x150 [mdd] 2012-09-18 14:13:37 [&lt;ffffffffa0eef9b1&gt;] __mdd_index_insert+0x51/0x1f0 [mdd] 2012-09-18 14:13:37 [&lt;ffffffffa0ef55e3&gt;] mdd_create+0x19a3/0x20c0 [mdd] 2012-09-18 14:13:37 [&lt;ffffffffa03aa5b1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 2012-09-18 14:13:37 [&lt;ffffffffa06a4637&gt;] cml_create+0x97/0x250 [cmm] 2012-09-18 14:13:37 [&lt;ffffffffa0f8bb9f&gt;] mdt_reint_open+0x108f/0x18a0 [mdt] 2012-09-18 14:13:37 [&lt;ffffffffa0f75151&gt;] mdt_reint_rec+0x41/0xe0 [mdt] 2012-09-18 14:13:37 [&lt;ffffffffa0f6e9aa&gt;] mdt_reint_internal+0x50a/0x810 [mdt] 2012-09-18 14:13:37 [&lt;ffffffffa0f6ef7d&gt;] mdt_intent_reint+0x1ed/0x500 [mdt] 2012-09-18 14:13:37 [&lt;ffffffffa0f6b191&gt;] mdt_intent_policy+0x371/0x6a0 [mdt] 2012-09-18 14:13:37 [&lt;ffffffffa0859881&gt;] ldlm_lock_enqueue+0x361/0x8f0 [ptlrpc] 2012-09-18 14:13:37 [&lt;ffffffffa08819bf&gt;] ldlm_handle_enqueue0+0x48f/0xf70 [ptlrpc] 2012-09-18 14:13:37 [&lt;ffffffffa0f6b506&gt;] mdt_enqueue+0x46/0x130 [mdt] 2012-09-18 14:13:37 [&lt;ffffffffa0f62802&gt;] mdt_handle_common+0x922/0x1740 [mdt] 2012-09-18 14:13:37 [&lt;ffffffffa0f636f5&gt;] mdt_regular_handle+0x15/0x20 [mdt] 2012-09-18 14:13:37 [&lt;ffffffffa08b199d&gt;] ptlrpc_server_handle_request+0x40d/0xea0 [ptlrpc] 2012-09-18 14:13:37 [&lt;ffffffffa08a8f37&gt;] ? ptlrpc_wait_event+0xa7/0x2a0 [ptlrpc] 2012-09-18 14:13:37 [&lt;ffffffffa03aa5b1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 2012-09-18 14:13:37 [&lt;ffffffff810533f3&gt;] ? __wake_up+0x53/0x70 2012-09-18 14:13:37 [&lt;ffffffffa08b2f89&gt;] ptlrpc_main+0xb59/0x1860 [ptlrpc] 2012-09-18 14:13:37 [&lt;ffffffffa08b2430&gt;] ? ptlrpc_main+0x0/0x1860 [ptlrpc] 2012-09-18 14:13:37 [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 2012-09-18 14:13:37 [&lt;ffffffffa08b2430&gt;] ? ptlrpc_main+0x0/0x1860 [ptlrpc] 2012-09-18 14:13:37 [&lt;ffffffffa08b2430&gt;] ? ptlrpc_main+0x0/0x1860 [ptlrpc] 2012-09-18 14:13:37 [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 2012-09-18 14:13:37 Code: 00 00 48 c7 c2 20 81 dd a0 4c 8b 45 a8 45 89 f1 48 c7 c6 e5 d2 dd a0 e8 9f 56 01 00 48 8b 55 c8 66 83 7a 04 00 0f 85 a9 fd ff ff &lt;0f&gt; 0b eb fe 49 8b 2012-09-18 14:13:37 LustreError: 4099:0:(fsfilt-ldiskfs.c:332:fsfilt_ldiskfs_start()) error starting handle <span class="code-keyword">for</span> op 10 (114 credits): rc -30 2012-09-18 14:13:37 LustreError: 4099:0:(llog_server.c:414:llog_origin_handle_cancel()) fsfilt_start_log() failed: -30 2012-09-18 14:13:37 LustreError: 4099:0:(llog_server.c:453:llog_origin_handle_cancel()) Cancel 0 of 121 llog-records failed: -30 2012-09-18 14:13:37 4c 24 40 49 8b bc 24 08 01 00 00 31 c0 4c 8b 2012-09-18 14:13:37 RIP [&lt;ffffffffa0db21d0&gt;] add_dirent_to_buf+0x4c0/0x530 [ldiskfs] 2012-09-18 14:13:37 RSP &lt;ffff880121e935f0&gt; 2012-09-18 14:13:38 Initializing cgroup subsys cpuset 2012-09-18 14:13:38 Initializing cgroup subsys cpu 2012-09-18 14:13:38 Linux version 2.6.32-279.5.1.el6_lustre.gb4cc145.x86 </pre> </div></div> <p>vmcore has been captured, is on brent.whamcloud.com ~/cliffw/lu1948/vmcore-rst6-c2.gz</p> <p>According to test result the corrupt directory comes from fdtree test. " 78644697: name L0F7, lck 0000000000000004, name_len=568931904". Hmm during fdtree test, each thread will only create files in its own directory, and only 10 files for each dir. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> \\srun -N $NODES -n $PROCS --wait=7200 -o fdtree.%t.out \$SWL/IO/fdtree.bash -l 3 -d 10 &gt;&amp; tmp_file </pre> </div></div> <p>So this problem should be unrelated with ldiskfs pdir patch. Now I suspect it might be related with mballoc, because we saw this during the test</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> "2012-09-17 17:49:57 LDISKFS-fs error (device md1): ldiskfs_mb_release_inode_pa: pa free mismatch: [pa ffff88031438e748] [phy 190752] [logic 608] [len 16] [free 13] [error 0] [inode 7340126] [freed 16] 2012-09-17 17:49:57 LDISKFS-fs error (device md1): ldiskfs_mb_release_inode_pa: free 16, pa_free 13 " </pre> </div></div> <p> though it happened after the fs turned to read-only.</p> <p>Add andreas to the ticket, in case there are some mballoc changes recently for ext4.</p> <p>There haven't been changes to mballoc, but there was a change to the symlink NUL termination recently. Does this workload create symlinks?</p> <p>Isn't osd_ldiskfs_write_record() writing one-byte off the buffer limit if write_NUL is true?</p> <p>fdtree does not create symlinks, which only includes mkdir, create, dd, unlink, rmdir. But SWL includes 5 tests, fdtree, simul, IOR, mirIO, mdtest. Simul definitely include create symlinks here. </p> <p>oh, if you mean <a href="https://jira.whamcloud.com/browse/LU-1540" title="e2fsck remove too many symlinks" class="issue-link" data-issue-key="LU-1540"><del>LU-1540</del></a>, which has been landed on 2_3, and already included in our test rpm here.</p> <p>This problem should be a duplicate with 1976, and Fang yong already provide a fix there, close this one.</p> <p>System crashed again w/liang's patch - dump taken<br/> stack is a bit messed up</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">2012-09-27 21:56:35 LustreError: 5611:0:(osd_handler.c:2343:osd_object_ref_del()) ASSERTION( inode-&gt;i_nlink &gt; 0 ) failed: 2012-09-27 21:56:35 LustreError: 5611:0:(osd_handler.c:2343:osd_object_ref_del()) LBUG 2012-09-27 21:56:35 Pid: 5611, comm: mdt00_015 2012-09-27 21:56:35 2012-09-27 21:56:35 Sep 27 21:56:35 Call Trace: 2012-09-27 21:56:35 hyperion-rst6 ke [&lt;ffffffffa0392905&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 2012-09-27 21:56:35 rnel: LustreErro [&lt;ffffffffa0392f17&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 2012-09-27 21:56:35 r: 5611:0:(osd_h [&lt;ffffffffa0a946a1&gt;] osd_object_ref_del+0x1d1/0x210 [osd_ldiskfs] 2012-09-27 21:56:35 andler.c:2343:os [&lt;ffffffffa0efa09d&gt;] mdo_ref_del+0xad/0xb0 [mdd] 2012-09-27 21:56:35 d_object_ref_del [&lt;ffffffffa0eff715&gt;] mdd_unlink+0x815/0xdb0 [mdd] 2012-09-27 21:56:35 ()) ASSERTION( i [&lt;ffffffffa09581e4&gt;] ? lustre_msg_get_versions+0xa4/0x120 [ptlrpc] 2012-09-27 21:56:35 node-&gt;i_nlink &gt; [&lt;ffffffffa08bd037&gt;] cml_unlink+0x97/0x200 [cmm] 2012-09-27 21:56:35 0 ) failed: 2012-09-27 21:56:35 Sep [&lt;ffffffffa0f83ddf&gt;] ? mdt_version_get_save+0x8f/0xd0 [mdt] 2012-09-27 21:56:35 27 21:56:35 hyp [&lt;ffffffffa0f84454&gt;] mdt_reint_unlink+0x634/0x9e0 [mdt] 2012-09-27 21:56:35 erion-rst6 kerne [&lt;ffffffffa0f81151&gt;] mdt_reint_rec+0x41/0xe0 [mdt] 2012-09-27 21:56:35 l: LustreError: [&lt;ffffffffa0f7a9aa&gt;] mdt_reint_internal+0x50a/0x810 [mdt] 2012-09-27 21:56:35 5611:0:(osd_hand [&lt;ffffffffa0f7acf4&gt;] mdt_reint+0x44/0xe0 [mdt] 2012-09-27 21:56:35 ler.c:2343:osd_o [&lt;ffffffffa0f6e802&gt;] mdt_handle_common+0x922/0x1740 [mdt] 2012-09-27 21:56:35 bject_ref_del()) [&lt;ffffffffa0f6f6f5&gt;] mdt_regular_handle+0x15/0x20 [mdt] 2012-09-27 21:56:35 LBUG 2012-09-27 21:56:35 [&lt;ffffffffa0966b3c&gt;] ptlrpc_server_handle_request+0x41c/0xe00 [ptlrpc] 2012-09-27 21:56:35 [&lt;ffffffffa039365e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] 2012-09-27 21:56:35 [&lt;ffffffffa03a513f&gt;] ? lc_watchdog_touch+0x6f/0x180 [libcfs] 2012-09-27 21:56:35 [&lt;ffffffffa095df37&gt;] ? ptlrpc_wait_event+0xa7/0x2a0 [ptlrpc] 2012-09-27 21:56:35 [&lt;ffffffff810533f3&gt;] ? __wake_up+0x53/0x70 2012-09-27 21:56:35 [&lt;ffffffffa0968111&gt;] ptlrpc_main+0xbf1/0x19e0 [ptlrpc] 2012-09-27 21:56:35 [&lt;ffffffffa0967520&gt;] ? ptlrpc_main+0x0/0x19e0 [ptlrpc] 2012-09-27 21:56:35 [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 2012-09-27 21:56:35 [&lt;ffffffffa0967520&gt;] ? ptlrpc_main+0x0/0x19e0 [ptlrpc] 2012-09-27 21:56:35 [&lt;ffffffffa0967520&gt;] ? ptlrpc_main+0x0/0x19e0 [ptlrpc] 2012-09-27 21:56:35 [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 2012-09-27 21:56:35 2012-09-27 21:56:35 Kernel panic - not syncing: LBUG 2012-09-27 21:56:35 Pid: 5611, comm: mdt00_015 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.x86_64 #1 2012-09-27 21:56:35 Sep 27 21:56:35 Call Trace: 2012-09-27 21:56:35 hyperion-rst6 ke [&lt;ffffffff814fd58a&gt;] ? panic+0xa0/0x168 2012-09-27 21:56:35 rnel: Kernel pan [&lt;ffffffffa0392f6b&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] 2012-09-27 21:56:35 ic - not syncing [&lt;ffffffffa0a946a1&gt;] ? osd_object_ref_del+0x1d1/0x210 [osd_ldiskfs] 2012-09-27 21:56:35 : LBUG 2012-09-27 21:56:35 [&lt;ffffffffa0efa09d&gt;] ? mdo_ref_del+0xad/0xb0 [mdd] 2012-09-27 21:56:35 [&lt;ffffffffa0eff715&gt;] ? mdd_unlink+0x815/0xdb0 [mdd] 2012-09-27 21:56:35 [&lt;ffffffffa09581e4&gt;] ? lustre_msg_get_versions+0xa4/0x120 [ptlrpc] 2012-09-27 21:56:35 [&lt;ffffffffa08bd037&gt;] ? cml_unlink+0x97/0x200 [cmm] 2012-09-27 21:56:35 [&lt;ffffffffa0f83ddf&gt;] ? mdt_version_get_save+0x8f/0xd0 [mdt] 2012-09-27 21:56:35 [&lt;ffffffffa0f84454&gt;] ? mdt_reint_unlink+0x634/0x9e0 [mdt] 2012-09-27 21:56:35 [&lt;ffffffffa0f81151&gt;] ? mdt_reint_rec+0x41/0xe0 [mdt] 2012-09-27 21:56:35 [&lt;ffffffffa0f7a9aa&gt;] ? mdt_reint_internal+0x50a/0x810 [mdt] 2012-09-27 21:56:35 [&lt;ffffffffa0f7acf4&gt;] ? mdt_reint+0x44/0xe0 [mdt] 2012-09-27 21:56:35 [&lt;ffffffffa0f6e802&gt;] ? mdt_handle_common+0x922/0x1740 [mdt] 2012-09-27 21:56:36 [&lt;ffffffffa0f6f6f5&gt;] ? mdt_regular_handle+0x15/0x20 [mdt] 2012-09-27 21:56:36 [&lt;ffffffffa0966b3c&gt;] ? ptlrpc_server_handle_request+0x41c/0xe00 [ptlrpc] 2012-09-27 21:56:36 [&lt;ffffffffa039365e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] 2012-09-27 21:56:36 [&lt;ffffffffa03a513f&gt;] ? lc_watchdog_touch+0x6f/0x180 [libcfs] 2012-09-27 21:56:36 [&lt;ffffffffa095df37&gt;] ? ptlrpc_wait_event+0xa7/0x2a0 [ptlrpc] 2012-09-27 21:56:36 [&lt;ffffffff810533f3&gt;] ? __wake_up+0x53/0x70 2012-09-27 21:56:36 [&lt;ffffffffa0968111&gt;] ? ptlrpc_main+0xbf1/0x19e0 [ptlrpc] 2012-09-27 21:56:36 [&lt;ffffffffa0967520&gt;] ? ptlrpc_main+0x0/0x19e0 [ptlrpc] 2012-09-27 21:56:36 [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 2012-09-27 21:56:36 [&lt;ffffffffa0967520&gt;] ? ptlrpc_main+0x0/0x19e0 [ptlrpc] 2012-09-27 21:56:36 [&lt;ffffffffa0967520&gt;] ? ptlrpc_main+0x0/0x19e0 [ptlrpc] 2012-09-27 21:56:36 [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 2012-09-27 21:56:36 Initializing cgroup subsys cpuset 2012-09-27 21:56:36 Initializing cgroup subsys cpu </pre> </div></div> <p>sigh, I would say this is a different bug, I just found it, it's <a href="https://jira.whamcloud.com/browse/LU-1951" title="SWL: osd_handler.c:2343:osd_object_ref_del()) ASSERTION( inode-&gt;i_nlink &gt; 0 ) failed:" class="issue-link" data-issue-key="LU-1951"><del>LU-1951</del></a></p> <p>duplicate of <a href="https://jira.whamcloud.com/browse/LU-2041" title="SWL ldiskfs_add_entry: bad entry in directory #127928380: rec_len is smaller than minimal" class="issue-link" data-issue-key="LU-2041"><del>LU-2041</del></a></p> <p>Hit again in most recent SWL test </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Oct 8 20:55:58 hyperion-rst6 kernel: Lustre: 4204:0:(service.c:2105:ptlrpc_handle_rs()) All locks stolen from rs ffff88012a6c9000 x1415301393993750.t4460647409 o0 NID 192.168.116.125@o2ib1 Oct 8 20:58:24 hyperion-rst6 kernel: LDISKFS-fs error (device md1): ldiskfs_add_entry: bad entry in directory #40370421: rec_len is smaller than minimal - block=20251025offset=504(504), inode=40380706, rec_len=0, name_len=4 Oct 8 20:58:24 hyperion-rst6 kernel: Aborting journal on device md1-8. Oct 8 20:58:24 hyperion-rst6 kernel: LDISKFS-fs error (device md1): ldiskfs_journal_start_sb: Oct 8 20:58:24 hyperion-rst6 kernel: LDISKFS-fs error (device md1): ldiskfs_journal_start_sb: Detected aborted journal Oct 8 20:58:24 hyperion-rst6 kernel: LDISKFS-fs (md1): Remounting filesystem read-only Oct 8 20:58:24 hyperion-rst6 kernel: LDISKFS-fs (md1): Remounting filesystem read-only Oct 8 20:58:24 hyperion-rst6 kernel: LDISKFS-fs error (device md1) in iam_txn_add: Journal has aborted Oct 8 20:58:24 hyperion-rst6 kernel: LustreError: 4885:0:(osd_io.c:1014:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30 </pre> </div></div> Development Rank 1|hzvgi7: Rank (Obsolete) 6323 Severity