https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2041 Lustre <p>Running SWL test, after ~6 hours seeing ldiskfs failure and EROFS on the MDT.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Sep 27 20:13:20 hyperion-rst6 kernel: Lustre: 4918:0:(mdt_recovery.c:622:mdt_steal_ack_locks()) Stealing 1 locks from rs ffff880108e55000 x1414295525519816.t4537566185 o0 NID 192.168.116.112@o2ib1 Sep 27 20:13:20 hyperion-rst6 kernel: Lustre: 4233:0:(service.c:2105:ptlrpc_handle_rs()) All locks stolen from rs ffff880108e55000 x1414295525519816.t4537566185 o0 NID 192.168.116.112@o2ib1 Sep 27 20:16:39 hyperion-rst6 kernel: LDISKFS-fs error (device md1): ldiskfs_add_entry: Sep 27 20:16:39 hyperion-rst6 kernel: LDISKFS-fs error (device md1): ldiskfs_add_entry: bad entry in directory #127928380: rec_len is smaller than minimal - block=64030811offset=536(536), inode=127946027, rec_len=0, name_len=4 Sep 27 20:16:39 hyperion-rst6 kernel: Aborting journal on device md1-8. Sep 27 20:16:39 hyperion-rst6 kernel: LDISKFS-fs error (device md1): ldiskfs_journal_start_sb: Detected aborted journal Sep 27 20:16:39 hyperion-rst6 kernel: LDISKFS-fs (md1): Remounting filesystem read-only Sep 27 20:16:39 hyperion-rst6 kernel: LDISKFS-fs error (device md1): ldiskfs_journal_start_sb: Detected aborted journal Sep 27 20:16:39 hyperion-rst6 kernel: LustreError: 4774:0:(fsfilt-ldiskfs.c:332:fsfilt_ldiskfs_start()) error starting handle <span class="code-keyword">for</span> op 10 (114 credits): rc -30 Sep 27 20:16:39 hyperion-rst6 kernel: LustreError: 4774:0:(llog_server.c:414:llog_origin_handle_cancel()) fsfilt_start_log() failed: -30 Sep 27 20:16:39 hyperion-rst6 kernel: LustreError: 4774:0:(llog_server.c:453:llog_origin_handle_cancel()) Cancel 0 of 121 llog-records failed: -30 Sep 27 20:16:39 hyperion-rst6 kernel: LustreError: 4774:0:(fsfilt-ldiskfs.c:332:fsfilt_ldiskfs_start()) error starting handle <span class="code-keyword">for</span> op 10 (114 credits): rc -30 Sep 27 20:16:39 hyperion-rst6 kernel: LustreError: 4774:0:(llog_server.c:414:llog_origin_handle_cancel()) fsfilt_start_log() failed: -30 Sep 27 20:16:39 hyperion-rst6 kernel: LustreError: 4774:0:(llog_server.c:453:llog_origin_handle_cancel()) Cancel 0 of 121 llog-records failed: -30 Sep 27 20:16:39 hyperion-rst6 kernel: LDISKFS-fs (md1): bad entry in directory #127928380: rec_len is smaller than minimal - block=64030811offset=536(536), inode=127946027, rec_len=0, name_len=4 Sep 27 20:16:39 hyperion-rst6 kernel: LDISKFS-fs error (device md1) in iam_txn_add: Journal has aborted Sep 27 20:16:39 hyperion-rst6 kernel: LustreError: 4865:0:(osd_io.c:1014:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30 Sep 27 20:16:39 hyperion-rst6 kernel: LustreError: 4865:0:(osd_handler.c:894:osd_trans_stop()) Failure in transaction hook: -30 Sep 27 20:16:39 hyperion-rst6 kernel: LustreError: 4865:0:(osd_handler.c:899:osd_trans_stop()) Failure to stop transaction: -30 Sep 27 20:16:39 hyperion-rst6 kernel: LDISKFS-fs error (device md1) in iam_txn_add: Journal has aborted Sep 27 20:16:39 hyperion-rst6 kernel: LustreError: 4731:0:(osd_io.c:1014:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30 Sep 27 20:16:39 hyperion-rst6 kernel: LustreError: 4731:0:(osd_handler.c:894:osd_trans_stop()) Failure in transaction hook: -30 Sep 27 20:16:39 hyperion-rst6 kernel: LustreError: 4731:0:(osd_handler.c:899:osd_trans_stop()) Failure to stop transaction: -30 Sep 27 20:16:39 hyperion-rst6 kernel: Remounting filesystem read-only Sep 27 20:16:39 hyperion-rst6 kernel: LDISKFS-fs error (device md1) in iam_txn_add: Journal has aborted </pre> </div></div> <p>Ran fsck on device, recovered</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LDISKFS-fs (md1): mounted filesystem with ordered data mode. quota=off. Opts: LDISKFS-fs (md1): mounted filesystem with ordered data mode. quota=off. Opts: Lustre: MGC192.168.127.6@o2ib: Reactivating <span class="code-keyword">import</span> Lustre: lustre-MDT0000: used disk, loading Lustre: 5311:0:(ldlm_lib.c:2139:target_recovery_init()) RECOVERY: service lustre-MDT0000, 91 recoverable clients, last_transno 4541171590 Lustre: lustre-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450 Lustre: 4326:0:(mgc_request.c:1534:mgc_process_recover_log()) <span class="code-object">Process</span> recover log lustre-mdtir error -22 Lustre: 4326:0:(mgc_request.c:1534:mgc_process_recover_log()) Skipped 2 previous similar messages Lustre: lustre-MDT0000: Will be in recovery <span class="code-keyword">for</span> at least 2:30, or until 91 clients reconnect LustreError: 5315:0:(ldlm_lib.c:1717:check_for_next_transno()) lustre-MDT0000: waking <span class="code-keyword">for</span> gap in transno, VBR is OFF (skip: 4541171591, ql: 62, comp: 29, conn: 91, next: 4541177085, last_committed: 4541171590) LustreError: 5315:0:(mds_lov.c:351:mds_lov_update_objids()) Unexpected gap in objids LustreError: 5315:0:(ldlm_lib.c:1717:check_for_next_transno()) lustre-MDT0000: waking <span class="code-keyword">for</span> gap in transno, VBR is OFF (skip: 4541177086, ql: 62, comp: 29, conn: 91, next: 4541181324, last_committed: 4541171590) Lustre: lustre-MDT0000: disconnecting 1 stale clients Lustre: lustre-MDT0000: Recovery over after 1:24, of 91 clients 90 recovered and 1 was evicted. Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0028_UUID now active, resetting orphans </pre> </div></div> <p>Will continue to run, see if we get repeat.</p> SWL - Hyperion/LLNL LU-2041

SWL ldiskfs_add_entry: bad entry in directory #127928380: rec_len is smaller than minimal

Bug Critical Closed Fixed nasf Cliff White Fri, 28 Sep 2012 00:18:31 +0000 Thu, 18 Oct 2012 14:06:22 +0000 Thu, 18 Oct 2012 14:06:22 +0000 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.3.0 Lustre 2.4.0 0 7 <p>Fanyong</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>It is really strange failure. Several possible reasons:</p> <p>1) The block was reused by others, and the dentry was re-written.</p> <p>Recently, only OI shrink patch may affect the block allocation, but such patch only affect the blocks belong to OI files, those blocks can never be reused by other non-OI directories, in spite of whether the OI shrink patch added or not. So it should not related such patch.</p> <p>On the other hand, if the block is re-written by others, then it is quite possible that the whole entry was re-wrriten, but not just the "rec_len" itself. But from the log, the "ino#" and "name_len" are reasonable values. So the blocks should not be re-written.</p> <p>2) ldiskfs internal bugs.<br/> There may some corner cases in ldiskfs leaf nodes split improperly set wrong "rec_len". We can add some debug patch for checking.</p> <p>3) Hardware issues.</p> <p>So if we can reproduce the failure, then it may be caused by 2), otherwise, may be 3).</p> <p>This is the patch for 2):</p> <p><a href="http://review.whamcloud.com/#change,4118" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4118</a></p> <p>Sorry for stepping in here. Clearly 1948 is not being fixed with the patch of 1976. I just re-investigated the result Cliff got on 1948. It seems to me two threads concurrently access the dir. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Thread A adds entry to dir 78644697, and met zero rec_len here.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 2012-09-18 14:13:37 LDISKFS-fs error (device md1): add_dirent: 2012-09-18 14:13:37 LDISKFS-fs error (device md1): ldiskfs_add_entry: bad entry in directory #78644697: rec_len is smaller than minimal - block=39336951offset=152(152), inode=78655820, rec_len=0, name_len=4 2012-09-18 14:13:37 Aborting journal on device md1-8. </pre> </div></div> <p>Thread B hit BUG_ON in the later place, which I added for debugging the problem, </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 1930 @@ -1387,7 +1751,17 @@ 1931 » if (de-&gt;inode) { 1932 » » struct ext4_dir_entry_2 *de1 = (struct ext4_dir_entry_2 *)((char *)de + nlen); 1933 » » de1-&gt;rec_len = ext4_rec_len_to_disk(rlen - nlen, blocksize); 1934 +» » if (de1-&gt;rec_len == 0) { 1935 +» » » __ext4_error(dir-&gt;i_sb, function, "%lu: name %s, lck %p, name_len=%d\n", 1936 +» » » » dir-&gt;i_ino, name, name_len); 1937 +» » » BUG_ON(de1-&gt;rec_len == 0); 1938 +» » } 1939 » » de-&gt;rec_len = ext4_rec_len_to_disk(nlen, blocksize); 1940 +» » if (de-&gt;rec_len == 0) { 1941 +» » » __ext4_error(dir-&gt;i_sb, function, "%lu: name %s, lck %p name_len=%d\n", 1942 +» » » » dir-&gt;i_ino, name, name_len); 1943 +» » » BUG_ON(de-&gt;rec_len == 0); 1944 +» » } 1945 » » de = de1; 1946 » } 1947 » de-&gt;file_type = EXT4_FT_UNKNOWN; </pre> </div></div> <p>The console message for hitting the BUG.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 2012-09-18 14:13:37 78644697: name L0F7, name_len=4 2012-09-18 14:13:37 2012-09-18 14:13:37 ------------[ cut here ]------------ 2012-09-18 14:13:37 kernel BUG at /var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el6/ib_stack/inkernel/BUILD/BUILD/lustre-ldiskfs-3.3.0/ldiskfs/namei.c:1757! 2012-09-18 14:13:37 invalid opcode: 0000 [#1] SMP 2012-09-18 14:13:37 last sysfs file: /sys/devices/pci0000:00/0000:00:09.0/0000:05:00.0/infiniband_mad/umad0/port 2012-09-18 14:13:37 CPU 7 ...... </pre> </div></div> <p>If thread A and B are serialized for this function, both of them should fail at ldiskfs_check_dir_entry. So it still seems ldiskfs pdir patch problem. Liang, Could you please have a look again for this one. </p> <p>And also for this test, each dir only includes 10 files, so it should not involve index splitting IMHO, all dir are just single block dir. </p> <p>for directory w/o indices, PDO lock equals to a RW semaphore which is exclusive for writing, which means PDO is actually <em>disabled</em> for small directory like this. But I will check my code again...</p> <p>Nasf, I would suggest to run with wangdi's patch on <a href="https://jira.whamcloud.com/browse/LU-1948" title="ldiskfs - MDS goes read-only (SWL)" class="issue-link" data-issue-key="LU-1948"><del>LU-1948</del></a> to gather more information instead of yours patch, actually I just added a little more debug information to Di's patch: <a href="http://review.whamcloud.com/#change,4020" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4020</a><br/> How do you think?</p> <p>Di, why you think the "If thread A and B are serialized for this function, both of them should fail at ldiskfs_check_dir_entry" ?</p> <p>If thread A call add_dirent_to_buf() with NULL "de", it will call ldiskfs_check_dir_entry(); If thread B call ldiskfs_check_dir_entry() with non-NULL "de", it will not call ldiskfs_check_dir_entry().</p> <p>So thread A and thread B may fail at different positions. Maybe I miss something, please correct me.</p> <p>Please see the trace on <a href="https://jira.whamcloud.com/browse/LU-1948" title="ldiskfs - MDS goes read-only (SWL)" class="issue-link" data-issue-key="LU-1948"><del>LU-1948</del></a></p> <p>2012-09-18 14:13:37 <span class="error">&#91;&lt;ffffffffa0db5c5d&gt;&#93;</span> ldiskfs_add_entry+0xcd/0x500 <span class="error">&#91;ldiskfs&#93;</span><br/> 2012-09-18 14:13:37 <span class="error">&#91;&lt;ffffffffa1001689&gt;&#93;</span> __osd_ea_add_rec+0xb9/0x190 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> 2012-09-18 14:13:37 <span class="error">&#91;&lt;ffffffffa100dbeb&gt;&#93;</span> osd_index_ea_insert+0x21b/0x5e0 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> 2012-09-18 14:13:37 <span class="error">&#91;&lt;ffffffffa0eee977&gt;&#93;</span> __mdd_index_insert_only+0x147/0x150 <span class="error">&#91;mdd&#93;</span><br/> 2012-09-18 14:13:37 <span class="error">&#91;&lt;ffffffffa0eef9b1&gt;&#93;</span> __mdd_index_insert+0x51/0x1f0 <span class="error">&#91;mdd&#93;</span><br/> 2012-09-18 14:13:37 <span class="error">&#91;&lt;ffffffffa0ef55e3&gt;&#93;</span> mdd_create+0x19a3/0x20c0 <span class="error">&#91;mdd&#93;</span><br/> 2012-09-18 14:13:37 <span class="error">&#91;&lt;ffffffffa03aa5b1&gt;&#93;</span> ? libcfs_debug_msg+0x41/0x50 <span class="error">&#91;libcfs&#93;</span><br/> 2012-09-18 14:13:37 <span class="error">&#91;&lt;ffffffffa06a4637&gt;&#93;</span> cml_create+0x97/0x250 <span class="error">&#91;cmm&#93;</span><br/> 2012-09-18 14:13:37 <span class="error">&#91;&lt;ffffffffa0f8bb9f&gt;&#93;</span> mdt_reint_open+0x108f/0x18a0 <span class="error">&#91;mdt&#93;</span><br/> 2012-09-18 14:13:37 <span class="error">&#91;&lt;ffffffffa0f75151&gt;&#93;</span> mdt_reint_rec+0x41/0xe0 <span class="error">&#91;mdt&#93;</span></p> <p>Then in ldiskfs_add_entry, because there are only about 10 entries per dir, so it should not be dx dir, and the size should be == 4096. </p> <p> blocks = dir-&gt;i_size &gt;&gt; sb-&gt;s_blocksize_bits;<br/> for (block = 0; block &lt; blocks; block++) {<br/> bh = ldiskfs_bread(handle, dir, block, 0, &amp;retval);<br/> if(!bh)<br/> return retval;<br/> retval = add_dirent_to_buf(handle, dentry, inode, NULL, bh); &lt;--- it should go into add_dirent_to_buf here, since the initial size should be 4096, and add 10 entries should not be over 4096.</p> <p> if (retval != -ENOSPC) </p> { brelse(bh); return retval; } <p> if (blocks == 1 &amp;&amp; !dx_fallback &amp;&amp;<br/> LDISKFS_HAS_COMPAT_FEATURE(sb, LDISKFS_FEATURE_COMPAT_DIR_INDEX))<br/> return make_indexed_dir(handle, dentry, inode, bh);<br/> brelse(bh);<br/> }<br/> bh = ldiskfs_append(handle, dir, &amp;block, &amp;retval);<br/> if (!bh)<br/> return retval;<br/> de = (struct ldiskfs_dir_entry_2 *) bh-&gt;b_data;<br/> de-&gt;inode = 0;<br/> de-&gt;rec_len = ldiskfs_rec_len_to_disk(blocksize, blocksize);<br/> retval = add_dirent_to_buf(handle, dentry, inode, de, bh); &lt;---- even it goes into from here, de-&gt;inode = 0 make it impossible to get to the BUG_ON place.</p> <p>fdtree will actually create 10 subdirs and 10 files, so it has 20 entries, but the directory is still plain w/o index, I tried this on my local workstation, and verified this.<br/> The question is, why we can only see this while running fdtree, although logs on this ticket didn't print out filename, but because namelen is still 4 which matches namelen of fdtree, so I guess the directory is still created by fdtree</p> <p>I checked PDO patch again but found nothing, because directory created by fdtree only has 20 entries and it's still a plain directory w/o index, I didn't change this part at all, also, directory created by fdtree has hostname+pid as part directory name, which means directory should be unique for each thread, if there's a bug to mess up file creation for a single thread, we might have seen it months ago because PDO is a 2.2 feature.</p> <p>However, I can't explain why we can see two failures in different places (wangdi's previous comment), the only possibility comes to my mind is OSD created two instances of htree_lock_head for the directory, or even created two lu_object for the same directory in rare case. </p> <p>The patch I posted in previous comment should give information to verify this (<a href="http://review.whamcloud.com/#change,4020" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4020</a>, build is <a href="http://build.whamcloud.com/job/lustre-reviews/9547/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/9547/</a>), so we might have to wait for the other try with this patch.</p> <p>I rebased my debug patch and make it depend on patch of <a href="https://jira.whamcloud.com/browse/LU-1951" title="SWL: osd_handler.c:2343:osd_object_ref_del()) ASSERTION( inode-&gt;i_nlink &gt; 0 ) failed:" class="issue-link" data-issue-key="LU-1951"><del>LU-1951</del></a>, the build is ready:<br/> <a href="http://build.whamcloud.com/job/lustre-reviews/9573/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/9573/</a></p> <p>I merged fix of <a href="https://jira.whamcloud.com/browse/LU-1951" title="SWL: osd_handler.c:2343:osd_object_ref_del()) ASSERTION( inode-&gt;i_nlink &gt; 0 ) failed:" class="issue-link" data-issue-key="LU-1951"><del>LU-1951</del></a>, fix of <a href="https://jira.whamcloud.com/browse/LU-2041" title="SWL ldiskfs_add_entry: bad entry in directory #127928380: rec_len is smaller than minimal" class="issue-link" data-issue-key="LU-2041"><del>LU-2041</del></a> and debug patch of <a href="https://jira.whamcloud.com/browse/LU-1948" title="ldiskfs - MDS goes read-only (SWL)" class="issue-link" data-issue-key="LU-1948"><del>LU-1948</del></a> together (<a href="http://review.whamcloud.com/#change,4153" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4153</a>) , here is the build of it: <a href="http://build.whamcloud.com/job/lustre-reviews/9619/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/9619/</a> , I think it's the next thing we should try on Hyperion, Cliff, could you please try it? Thanks.</p> <p>also add another fix from wangdi to the debug patch, the latest build is: <a href="http://build.whamcloud.com/job/lustre-reviews/9641/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/9641/</a></p> <p>just posted a patch : <a href="http://review.whamcloud.com/#change,4186" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4186</a><br/> It fixed a corner case in osd_it_ea_rec() which might cause inconsistent OI cache, I'm not 100% sure it's fix of this bug, but at least last time SWL ran much longer than before w/o seeing this bug after applying this patch.</p> <p>Crashed.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Oct 5 18:27:58 hyperion-rst6 kernel: LDISKFS-fs Dumping htree_lock ffff88011e8db200, mode 3, head ffff880139a39180 Oct 5 18:27:58 hyperion-rst6 kernel: granted mode: 3:1 Oct 5 18:27:58 hyperion-rst6 kernel: LDISKFS-fs error (device md1): add_dirent(de1): dir(134759485, dx: 1, nblks: 33) de(mdtest.16.41 : 12), off 2376 rlen 40 nlen 40 Oct 5 18:27:58 hyperion-rst6 kernel: LDISKFS-fs error (device md1): ldiskfs_dx_find_entry: bad entry in directory #134759485: rec_len is smaller than minimal - block=67436580offset=2416(59760), inode=134767633, rec_len=0, name_len=5 Oct 5 18:27:58 hyperion-rst6 kernel: Aborting journal on device md1-8. Oct 5 18:27:58 hyperion-rst6 kernel: LustreError: 4331:0:(osd_io.c:1014:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30 Oct 5 18:27:58 hyperion-rst6 kernel: LustreError: 4331:0:(osd_handler.c:894:osd_trans_stop()) Failure in transaction hook: -30 Oct 5 18:27:58 hyperion-rst6 kernel: LDISKFS-fs error (device md1) in osd_trans_stop: Journal has aborted Oct 5 18:27:58 hyperion-rst6 kernel: LDISKFS-fs (md1): Remounting filesystem read-only Oct 5 18:27:58 hyperion-rst6 kernel: LDISKFS-fs (md1): Remounting filesystem read-only Oct 5 18:27:58 hyperion-rst6 kernel: LustreError: 4331:0:(osd_handler.c:899:osd_trans_stop()) Failure to stop transaction: -30 Oct 5 18:27:58 hyperion-rst6 kernel: Oct 5 18:27:58 hyperion-rst6 kernel: ------------[ cut here ]------------ Oct 5 18:27:58 hyperion-rst6 kernel: kernel BUG at /<span class="code-keyword">var</span>/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el6/ib_stack/inkernel/BUILD/BUILD/lustre-ldiskfs-3.3.0/ldiskfs/namei.c:1757! Oct 5 18:27:58 hyperion-rst6 kernel: invalid opcode: 0000 [#1] SMP Oct 5 18:27:58 hyperion-rst6 kernel: last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map Oct 5 18:27:58 hyperion-rst6 kernel: CPU 5 Oct 5 18:27:59 hyperion-rst6 kernel: Modules linked in: cmm(U) osd_ldiskfs(U) mdt(U) mdd(U) mds(U) fsfilt_ldiskfs(U) exportfs mgs(U) mgc(U) lustre(U) lquota(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ldiskfs(U) mbcache jbd2 zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa mlx4_ib ib_mad ib_core dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun kvm raid0 sg sr_mod cdrom sd_mod crc_t10dif dcdbas serio_raw ata_generic pata_acpi ata_piix iTCO_wdt iTCO_vendor_support mptsas mptscsih mptbase scsi_transport_sas i7core_edac edac_core ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc mlx4_en mlx4_core bnx2 [last unloaded: scsi_wait_scan] Oct 5 18:27:59 hyperion-rst6 kernel: Oct 5 18:27:59 hyperion-rst6 kernel: Pid: 6331, comm: mdt03_016 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.gb16fe80.x86_64 #1 Dell Inc. PowerEdge R610/0K399H Oct 5 18:27:59 hyperion-rst6 kernel: RIP: 0010:[&lt;ffffffffa0726502&gt;] [&lt;ffffffffa0726502&gt;] add_dirent_to_buf+0x552/0x6a0 [ldiskfs] Oct 5 18:27:59 hyperion-rst6 kernel: RSP: 0018:ffff880159a17480 EFLAGS: 00010246 Oct 5 18:27:59 hyperion-rst6 kernel: RAX: ffff8801eea83800 RBX: ffff880313470948 RCX: ffff880313470970 Oct 5 18:27:59 hyperion-rst6 kernel: RDX: ffff88031833f400 RSI: 0000000000000046 RDI: ffff8801eea83c00 Oct 5 18:27:59 hyperion-rst6 kernel: RBP: ffff880159a17550 R08: 0000000000000000 R09: 0000000000000000 Oct 5 18:27:59 hyperion-rst6 kernel: R10: 0000000000000001 R11: 0000000000000000 R12: ffff880146a44250 Oct 5 18:27:59 hyperion-rst6 kernel: R13: 0000000000000948 R14: 000000000000000c R15: 0000000000000028 Oct 5 18:27:59 hyperion-rst6 kernel: FS: 00002aaaab47e700(0000) GS:ffff880028240000(0000) knlGS:0000000000000000 Oct 5 18:27:59 hyperion-rst6 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Oct 5 18:27:59 hyperion-rst6 kernel: CR2: 00002aaaab34230c CR3: 0000000001a85000 CR4: 00000000000006e0 Oct 5 18:27:59 hyperion-rst6 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Oct 5 18:27:59 hyperion-rst6 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Oct 5 18:27:59 hyperion-rst6 kernel: Stack: Oct 5 18:27:59 hyperion-rst6 kernel: ffffc9002ae7e1f0 000000000000000c ffff880100000948 0000000000000028 Oct 5 18:27:59 hyperion-rst6 kernel: &lt;d&gt; 0000000000000028 00000000ecc23e5e ffff8801abec6080 ffffffff080863e4 Oct 5 18:27:59 hyperion-rst6 kernel: &lt;d&gt; 0000000000021000 ffffffffa074720c ffff88011e8db200 ffff880113f7cc18 Oct 5 18:27:59 hyperion-rst6 kernel: Call Trace: Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0728999&gt;] ldiskfs_dx_add_entry+0x169/0x7e0 [ldiskfs] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa07433f1&gt;] ? ldiskfs_xattr_set_handle+0x151/0x560 [ldiskfs] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa072929d&gt;] ldiskfs_add_entry+0x28d/0x500 [ldiskfs] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0f13689&gt;] __osd_ea_add_rec+0xb9/0x190 [osd_ldiskfs] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0f1fc1b&gt;] osd_index_ea_insert+0x21b/0x5e0 [osd_ldiskfs] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0e21d2d&gt;] ? __mdd_xattr_set+0x25d/0x380 [mdd] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0e3a767&gt;] __mdd_index_insert_only+0x147/0x150 [mdd] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0e3b901&gt;] __mdd_index_insert+0x51/0x1f0 [mdd] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0e3bdf9&gt;] ? mdd_object_capa+0x29/0x1b0 [mdd] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0e416eb&gt;] mdd_create+0x1a0b/0x2180 [mdd] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0f19f9f&gt;] ? osd_xattr_get+0x9f/0x360 [osd_ldiskfs] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0f6d637&gt;] cml_create+0x97/0x250 [cmm] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0eaaddf&gt;] ? mdt_version_get_save+0x8f/0xd0 [mdt] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0ebeb9f&gt;] mdt_reint_open+0x108f/0x18a0 [mdt] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0e4728e&gt;] ? md_ucred+0x1e/0x60 [mdd] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0e8c235&gt;] ? mdt_ucred+0x15/0x20 [mdt] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0ea8151&gt;] mdt_reint_rec+0x41/0xe0 [mdt] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0ea19aa&gt;] mdt_reint_internal+0x50a/0x810 [mdt] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0ea1f7d&gt;] mdt_intent_reint+0x1ed/0x500 [mdt] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0e9e191&gt;] mdt_intent_policy+0x371/0x6a0 [mdt] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa090d881&gt;] ldlm_lock_enqueue+0x361/0x8f0 [ptlrpc] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa09359bf&gt;] ldlm_handle_enqueue0+0x48f/0xf70 [ptlrpc] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0e9e506&gt;] mdt_enqueue+0x46/0x130 [mdt] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0e95802&gt;] mdt_handle_common+0x922/0x1740 [mdt] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0e966f5&gt;] mdt_regular_handle+0x15/0x20 [mdt] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0965b3c&gt;] ptlrpc_server_handle_request+0x41c/0xe00 [ptlrpc] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa039365e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa03a514f&gt;] ? lc_watchdog_touch+0x6f/0x180 [libcfs] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa095cf37&gt;] ? ptlrpc_wait_event+0xa7/0x2a0 [ptlrpc] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffff810533f3&gt;] ? __wake_up+0x53/0x70 Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0967111&gt;] ptlrpc_main+0xbf1/0x19e0 [ptlrpc] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0966520&gt;] ? ptlrpc_main+0x0/0x19e0 [ptlrpc] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0966520&gt;] ? ptlrpc_main+0x0/0x19e0 [ptlrpc] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffffa0966520&gt;] ? ptlrpc_main+0x0/0x19e0 [ptlrpc] Oct 5 18:27:59 hyperion-rst6 kernel: [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 Oct 5 18:27:59 hyperion-rst6 kernel: Code: 54 24 18 48 8b 55 b8 48 89 c1 31 c0 48 89 14 24 48 c7 c2 38 c3 74 a0 e8 0d 55 01 00 48 8b 4d a8 66 83 79 04 00 0f 85 13 fd ff ff &lt;0f&gt; 0b eb fe 48 8b 7d 80 e8 d1 e7 01 00 49 8b bc 24 08 01 00 00 Oct 5 18:27:59 hyperion-rst6 kernel: RIP [&lt;ffffffffa0726502&gt;] add_dirent_to_buf+0x552/0x6a0 [ldiskfs] Oct 5 18:27:59 hyperion-rst6 kernel: RSP &lt;ffff880159a17480&gt; </pre> </div></div> <p>I don't think we dropped a vmcore on this one, still looking</p> <p>this time we crashed in mdtest, and the directory is a directory with indices, so it's more like a PDO issue, however:</p> <ul> <li>we can see in the log (granted mode: 3:1), which means the directory only granted one thread to enter the directory, but there are two threads hit different errors in different places, under the same directory (with same ino: 134759485)</li> </ul> <ul> <li>there is a entry has name_len=5, but mdtest will never create file with name_len=5 <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 5 18:27:58 hyperion-rst6 kernel: LDISKFS-fs error (device md1): ldiskfs_dx_find_entry: bad entry in directory #134759485: rec_len is smaller than minimal - block=67436580offset=2416(59760), inode=134767633, rec_len=0, name_len=5 </pre> </div></div></li> </ul> <p>Based on previous two things, it could be either:<br/> a) PDO lock totally screwed up locking; </p> <p>or </p> <p>b) two OSD objects with different FIDs point to a same inode (directory) for some unknown reason.</p> <p>I think the only choice now is disable several things one by one:</p> <ul> <li>OI cache</li> <li>OI shrinker</li> <li>PDO</li> </ul> <p>So I have to create different builds to try...</p> <p>latest build is here: <a href="http://build.whamcloud.com/job/lustre-reviews/9744/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/9744/</a><br/> major change in this build is disabled OI cache</p> <p>This patch fixes some potential OI mapping cache inconsistency issues:</p> <p><a href="http://review.whamcloud.com/#change,4207" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4207</a></p> <p>The latest build is here: <a href="http://build.whamcloud.com/job/lustre-reviews/9758" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/9758</a></p> <p>I merged fanyong's fix into this build. </p> <p>Dropping priority because landed for 2.3</p> <p>Patch for master here: <a href="http://review.whamcloud.com/#change,4208" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4208</a></p> <p>Please let me know if this needs to be reopened for more work.</p> Duplicate LU-2080 Development Rank 1|hzv487: Rank (Obsolete) 4217 Severity