https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16405 Lustre <p>In <a href="https://jira.whamcloud.com/browse/LU-10235" title="mkdir should check for directory existence on client before taking write lock" class="issue-link" data-issue-key="LU-10235"><del>LU-10235</del></a>, name lookup is moved before parent locking to avoid unnecessary lock revoke, and it's believed that backend filesystem will check name before entry insert, but some test shows this is not always true.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ls -lisad pan_* 162139365236040503 4 lrwxrwxrwx 1 rdx rd 65 Dec 8 01:51 pan_000 -&gt; /ec/fws5/sb/work/rd/cxsj/hvxz/MMSF/2017050100/longrange/an_centre 162139357216535883 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_001 162139341278207222 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_002 162139365689006218 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_003 162139362014851225 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_004 162139364397192377 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_005 162139362031645470 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_006 162139361696099316 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_007 162139361343777112 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_008 162139353626266212 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_009 162139350539298147 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_010 162139350539298147 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_010 ## Here 162139362769806682 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_011 162139353609530857 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_012 162139356226776195 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_013 162139346428878656 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_014 162139361343777140 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_015 162139366141987047 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_016 162139358508461812 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_017 162139353072661755 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_018 162139353072661755 4 drwxr-x--- 2 rdx rd 4096 Dec 8 01:51 pan_018 ## and there </pre> </div></div> <p>The problem is hit with single-block directories, so is not related to htree split or similar:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#&gt; debugfs -c /dev/mapper/vg_mdt0003_f02-mdt0003 debugfs 1.46.2.wc3 (18-Jun-2021) /dev/mapper/vg_mdt0003_f02-mdt0003: catastrophic mode - not reading inode or group bitmaps debugfs: cd REMOTE_PARENT_DIR/0x240007130:0x1:0x0/work/2017050100/longrange debugfs: ls [...snip] 1032403872 (44) psuhvxz14.noSnow 1032403874 (1020) psuhvxz14 1038197629 (36) pan_004 1032401372 (48) __ICMGG3.noSnow.tmp 1039177323 (36) pan_010 1039222784 (36) pan_010 1032971017 (48) targethvxz20170501002 1032401160 (36) pert_001 [...] debugfs: stat &lt;1039177323&gt; Inode: 1039177323 Type: directory Mode: 0750 Flags: 0x20000000 Generation: 3276743085 Version: 0x0000002f:d1139426 User: 388 Group: 1100 Project: 13 Size: 4096 File ACL: 0 Links: 2 Blockcount: 8 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x63914328:4490c0f0 -- Thu Dec 8 01:51:36 2022 atime: 0x63988ad8:00000000 -- Tue Dec 13 14:23:20 2022 mtime: 0x63914328:4490c0f0 -- Thu Dec 8 01:51:36 2022 crtime: 0x63914328:4490c0f0 -- Thu Dec 8 01:51:36 2022 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x24008e159:0x1e563:0x0] compat=0 incompat=0 linkea: idx=0 parent=[0x24008e347:0x14e34:0x0] name='pan_010' BLOCKS: (0):649617594 TOTAL: 1 debugfs: stat &lt;1039222784&gt; Inode: 1039222784 Type: directory Mode: 0750 Flags: 0x20000000 Generation: 3276743100 Version: 0x0000002f:d1139472 User: 388 Group: 1100 Project: 13 Size: 4096 File ACL: 0 Links: 2 Blockcount: 8 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x6391432b:00000000 -- Thu Dec 8 01:51:39 2022 atime: 0x63914328:450ad5a4 -- Thu Dec 8 01:51:36 2022 mtime: 0x6391432b:00000000 -- Thu Dec 8 01:51:39 2022 crtime: 0x63914328:450ad5a4 -- Thu Dec 8 01:51:36 2022 Size of extra inode fields: 32 Extended attributes: lma: fid=[0x24008e422:0x1fcab:0x0] compat=0 incompat=0 linkea: idx=0 parent=[0x24008e347:0x14e34:0x0] name='pan_010' debugfs: ls &lt;1039177323&gt; 1039177323 (12) . 1032400653 (4084) .. debugfs: ls &lt;1039222784&gt; 1039222784 (12) . 1032400653 (28) .. 1039271133 (40) ICMSHhvxzINIT 1039279349 (40) ICMGGhvxzINIT 1039281023 (3976) ggml199 </pre> </div></div> LU-16405

regression in create that may cause directory entries with the same name

Bug Critical Resolved Fixed Alex Zhuravlev Lai Siyao Thu, 15 Dec 2022 09:24:55 +0000 Mon, 28 Aug 2023 16:17:04 +0000 Thu, 27 Jul 2023 12:29:35 +0000 Lustre 2.16.0 0 9 <p>looks quite strange.. ext4_find_dest_de() checks full block for duplicates. a hole in pdirops locking? is it specific to some kernel?</p> <p>Alex, would be el7.9 kernel. </p> <p>Is there a smarter way to avoid this than reverting <a href="https://jira.whamcloud.com/browse/LU-10235" title="mkdir should check for directory existence on client before taking write lock" class="issue-link" data-issue-key="LU-10235"><del>LU-10235</del></a> ?</p> <p>It's not proved yet, we tried to reproduce and capture debug log, but couldn't reproduce.</p> <p>It is reverted on the client used on customer's production for last 1 month and a half.</p> <p>I am currently waiting for confirmation that the problem was not seen since then.</p> <p>I will report here when I get confirmation.</p> <p>If this was proved to be the reason, would there be a better way than reverting ?</p> <p>Looking at the original <a href="https://jira.whamcloud.com/browse/LU-10235" title="mkdir should check for directory existence on client before taking write lock" class="issue-link" data-issue-key="LU-10235"><del>LU-10235</del></a> patch, it seems possible to add some lightweight check <b>after</b> the parent is locked (if no entry was found during the unlocked lookup) to determine if the directory was modified between the time the lookup was done and when it was locked. We don't want to reintroduce the overhead of reverting the whole patch, or worse <b>always</b> do two lookups and make the code slower than it was before. </p> <p>In the use case addressed by the <a href="https://jira.whamcloud.com/browse/LU-10235" title="mkdir should check for directory existence on client before taking write lock" class="issue-link" data-issue-key="LU-10235"><del>LU-10235</del></a> patch (many threads opening the same file) this should not add any overhead, but in the concurrent directory modification case it would decide if another lookup was needed with the lock held. It <em>might</em> be possible to take and drop the object lock before the parent lock, and then determine if the object lock was cancelled (i.e. conflicting operation) to force another lookup under the lock.</p> <p>But directory time is not accurate enough to know whether parent was changed, because create only modifies parent directory time to seconds.</p> <p>I wasn't thinking of using ctime (unless it changes to use nanoseconds?), but rather the directory version (either i_version or another in-memory version counter).</p> <p>iirc, htree locks specific block to add a new direntry? I'd try to reproduce this with a change to ldiskfs_find_dest_de() searching through a whole block for the given name.</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50505" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50505</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16405" title="regression in create that may cause directory entries with the same name" class="issue-link" data-issue-key="LU-16405"><del>LU-16405</del></a> tests: a reproducer<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: db199a9398cb6b773ccd58505eb7ca7d43053f8b</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50521" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50521</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16405" title="regression in create that may cause directory entries with the same name" class="issue-link" data-issue-key="LU-16405"><del>LU-16405</del></a> osd: lookup cache<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a3d94d8abfb8c0d9320687be578073d3e3387fa6</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50521/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50521/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16405" title="regression in create that may cause directory entries with the same name" class="issue-link" data-issue-key="LU-16405"><del>LU-16405</del></a> osd: lookup cache<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 29f8eb2a67ba2806d91d93de1e82e05a63f76382</p> <p>Fix landed for 2.16</p> <blockquote><p>the patch(#50521) uses function obj_name2lu_name which was added in patch(<a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/43390/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/43390/</a>)</p></blockquote> <p>yes, that's to support filesystem encryption. if we don't need to support encryption in this version, then we don't need to call obd_name2lu_name() and can use the passed key directly. feel free to ask if you have more questions.</p> <blockquote><p> can you please port the patch to b_es5_2 please?</p></blockquote> <p>sure, will do</p> Related LU-10235 Development Rank 1|i0382v: Rank (Obsolete) 9223372036854775807 Severity