https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16610 Lustre <p>Directory corruption when running io500 test on openEuler 22.03:</p> <p>Client side log </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [openeuler@oe2203-test io500]$ sudo /io500.sh config-minimal.ini IO500 version io500-sc22_v2 (standard) [RESULT]       ior-easy-write        0.105593 GiB/s : time 338.211 seconds ERROR: open64(<span class="code-quote">"/mnt/lustre/datafiles/2023.02.14-10.12.17/mdtest-easy/test-dir.0-0/mdtest_tree.0.0/file.mdtest.1.85"</span>, 66, 0664) failed. Error: Read-only file system, (aiori-POSIX.c:569) -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode -1.</pre> </div></div> <p>  <br/> Server side log </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 9962.007724] LDISKFS-fs error (device dm-0): ldiskfs_find_dest_de:2412: inode #5767170: block 3771253: comm mdt00_000: bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=8, name_len=0, size=4096 [ 9962.051171] Aborting journal on device dm-0-8. [ 9962.058456] LDISKFS-fs (dm-0): Remounting filesystem read-only [ 9962.059877] LDISKFS-fs error (device dm-0) in iam_txn_add:547: Journal has aborted [ 9962.064365] LustreError: 11366:0:(osd_io.c:2222:osd_ldiskfs_write_record()) journal_get_write_access() returned error -30 [ 9962.066805] LustreError: 11366:0:(llog_cat.c:592:llog_cat_add_rec()) llog_write_rec -30: lh=00000000c04e4ff3 [ 9962.069137] LustreError: 11366:0:(tgt_lastrcvd.c:1326:tgt_add_reply_data()) lustre-MDT0000: can't update reply_data file: rc = -30 [ 9962.071742] LustreError: 11366:0:(osd_handler.c:2089:osd_trans_stop()) lustre-MDT0000: failed in transaction hook: rc = -30 [ 9962.074184] LustreError: 11366:0:(osd_handler.c:2099:osd_trans_stop()) lustre-MDT0000: failed to stop transaction: rc = -30 [ 9962.074274] LustreError: 11348:0:(osd_handler.c:1789:osd_trans_commit_cb()) transaction @0x00000000c73ec34c commit error: 2</pre> </div></div> <p> </p> openEuler 22.03 kernel: 5.10.0-60.79.0.103.oe2203.aarch64 LU-16610

ldiskfs_find_dest_de bad entry in directory when running io500 test

Bug Minor Resolved Fixed Xinliang Liu Xinliang Liu Thu, 2 Mar 2023 02:19:14 +0000 Wed, 22 Mar 2023 01:57:18 +0000 Wed, 22 Mar 2023 01:57:18 +0000 Lustre 2.16.0 0 7 <p>A similar issue also happens in running the io500 test. But the kernel version is different.</p> <p><a href="https://jira.whamcloud.com/browse/LU-12268" title="LDISKFS-fs error: ldiskfs_find_dest_de:2066: bad entry in directory: rec_len is smaller than minimal - offset=0( 0), inode=201, rec_len=0, name_len=0" class="issue-link" data-issue-key="LU-12268"><del>LU-12268</del></a></p> <p>Bisected the related commit/patch:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> f94c02917f1d ext4: avoid cycles in directory h-tree ( Which included in openEuler 22.03 LTS kernel kernel-5.10.0-60.58.0.86.oe2203) ldiskfs/kernel_patches/patches/oe2203/ext4-pdirop.patch (Lustre ldiskfs patch on ext4) </pre> </div></div> <p>Workaround:</p> <p>revert commit “f94c02917f1d ext4: avoid cycles in directory h-tree“ and update ext4-pdirop.patch.</p> <p>see io500 test suite running result, it is running ok with this workaround.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [openeuler@oe2203-test io500]$ sudo ./io500 config-minimal.ini IO500 version io500-sc22_v2 (standard) [RESULT]       ior-easy-write        0.103132 GiB/s : time 316.294 seconds [RESULT]    mdtest-easy-write        0.067036 kIOPS : time 301.645 seconds [      ]            timestamp        0.000000 kIOPS : time 0.000 seconds [RESULT]       ior-hard-write        0.101985 GiB/s : time 312.619 seconds [RESULT]    mdtest-hard-write        0.054293 kIOPS : time 301.826 seconds [RESULT]                 find        3.992785 kIOPS : time 9.124 seconds [RESULT]        ior-easy-read        0.023636 GiB/s : time 1380.092 seconds [RESULT]     mdtest-easy-stat        0.107839 kIOPS : time 187.558 seconds [RESULT]        ior-hard-read        0.022159 GiB/s : time 1438.550 seconds [RESULT]     mdtest-hard-stat        0.203911 kIOPS : time 81.015 seconds [RESULT]   mdtest-easy-delete        0.106105 kIOPS : time 190.760 seconds [RESULT]     mdtest-hard-read        0.065468 kIOPS : time 250.149 seconds [RESULT]   mdtest-hard-delete        0.103164 kIOPS : time 159.408 seconds [SCORE ] Bandwidth 0.048447 GiB/s : IOPS 0.147904 kiops : TOTAL 0.084649 The result files are stored in the directory: ./results/2023.02.22-01.44.09 [openeuler@oe2203-test io500]$ uname -r 5.10.0-60.79.0.103.oe2203.aarch64 </pre> </div></div> <p> </p> <p>The commit “f94c02917f1d ext4: avoid cycles in directory h-tree“ should be ok, we need to tune the ext4-pdirop.patch maybe.</p> <p>It seems this issue related to below two code parts:</p> <p>Part1 (introduced by  commit: f94c02917f1d ext4: avoid cycles in directory h-tree)</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> block = dx_get_block(at); <span class="code-keyword">for</span> (i = 0; i &lt;= level; i++) {     <span class="code-keyword">if</span> (blocks[i] == block) {         ext4_warning_inode(dir,             <span class="code-quote">"dx entry: tree cycle block %u points back to block %u"</span>,             blocks[level], block);         <span class="code-keyword">goto</span> fail;     } }</pre> </div></div> <p>Part2 (introduced by ext4-pdirop.patch)</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (indirect == level) { <span class="code-comment">/* the last index level */</span>     struct ext4_dir_lock_data *ld;     u64 myblock;    /* By <span class="code-keyword">default</span> we only lock DE-block, however, we will      * also lock the last level DX-block <span class="code-keyword">if</span>:      * a) there is hash collision      *    we will set DX-lock flag (a few lines below)      *    and redo to lock DX-block      *    see detail in dx_probe_hash_collision()      * b) it's a retry from splitting      *    we need to lock the last level DX-block so nobody      *    <span class="code-keyword">else</span> can split any leaf blocks under the same      *    DX-block, see detail in ext4_dx_add_entry()      */     <span class="code-keyword">if</span> (ext4_htree_dx_locked(lck)) {         /* DX-block is locked, just lock DE-block          * and <span class="code-keyword">return</span>          */         ext4_htree_spin_unlock(lck);         <span class="code-keyword">if</span> (!ext4_htree_safe_locked(lck))                 ext4_htree_de_lock(lck, frame-&gt;at); ...     <span class="code-keyword">if</span> (myblock == EXT4_HTREE_NODE_CHANGED) {         /* someone split <span class="code-keyword">this</span> DE-block before          * I locked it, I need to retry and lock          * valid DE-block          */         ext4_htree_de_unlock(lck);         <span class="code-keyword">continue</span>;     }     <span class="code-keyword">return</span> frame; }</pre> </div></div> <p>After putting part2 after part1, this issue gone.</p> <p>Xinliang, great debugging. </p> <p>Verified that rhel9.1 kernel kernel-5.14.0-162.12.1.el9_1 has no this issue.</p> <p>Xinliang, since this is a bug in the ldiskfs patch series for that kernel version, can you please submit a patch to update that series with the fix. </p> <p>Andreas, sure. Working on it.</p> <p>"xinliang &lt;xinliang.liu@linaro.org&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50192" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50192</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16610" title="ldiskfs_find_dest_de bad entry in directory when running io500 test" class="issue-link" data-issue-key="LU-16610"><del>LU-16610</del></a> ldiskfs: fix directory corruption on openeuler 22.03<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9280d8c880e534629df89b7d40e531297e701c99</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50192/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50192/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16610" title="ldiskfs_find_dest_de bad entry in directory when running io500 test" class="issue-link" data-issue-key="LU-16610"><del>LU-16610</del></a> ldiskfs: fix directory corruption on openeuler 22.03<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 85b76aa91a3999a325a9ef970f0cc8b6dd1cdda7</p> <p>Landed for 2.16</p> Related LU-12268 Development Rank 1|i03fhb: Rank (Obsolete) 9223372036854775807 Severity