https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1777 Lustre <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root]# /usr/src/lustre-release/lustre/tests/llmount.sh [root]# cd /mnt/lustre/ [root]# mkdir sanity [root]# chown sanity: sanity [root]# su sanity [sanity]$ pwd /mnt/lustre [sanity]$ sys_path2fid . [0x61ab:0xef3d87c8:0x0] [sanity]$ sys_rename sanity .lustre/fid/[0x61ab:0xef3d87c8:0x0]/sanity </pre> </div></div> <p>rename() wedges in lock_rename().</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>INFO: task sys_rename:2960 blocked for more than 120 seconds. "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. sys_rename D 0000000000000000 0 2960 2933 0x00000080 ffff88005cc37cf8 0000000000000082 ffff88005cc37d08 ffffffff81189a05 0000001000000000 ffff88007b01cb70 ffffffff8100bc0e ffff88005cc37cf8 ffff880062a67098 ffff88005cc37fd8 000000000000fb88 ffff880062a67098 Call Trace: [&lt;ffffffff81189a05&gt;] ? __link_path_walk+0x155/0x1030 [&lt;ffffffff8100bc0e&gt;] ? apic_timer_interrupt+0xe/0x20 [&lt;ffffffff8104f18b&gt;] ? mutex_spin_on_owner+0x9b/0xc0 [&lt;ffffffff814ff2fe&gt;] __mutex_lock_slowpath+0x13e/0x180 [&lt;ffffffff814ff19b&gt;] mutex_lock+0x2b/0x50 [&lt;ffffffff811878e3&gt;] lock_rename+0x73/0xe0 [&lt;ffffffff8118af83&gt;] sys_renameat+0x113/0x260 [&lt;ffffffff8119a470&gt;] ? mntput_no_expire+0x30/0x110 [&lt;ffffffff8117cb11&gt;] ? __fput+0x1a1/0x210 [&lt;ffffffff81142c7e&gt;] ? remove_vma+0x6e/0x90 [&lt;ffffffff810d6b12&gt;] ? audit_syscall_entry+0x272/0x2a0 [&lt;ffffffff815036de&gt;] ? do_page_fault+0x3e/0xa0 [&lt;ffffffff8118b0eb&gt;] sys_rename+0x1b/0x20 [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root]# pidof sys_rename 2960 [root]# cat /proc/2960/stack [&lt;ffffffff811878e3&gt;] lock_rename+0x73/0xe0 [&lt;ffffffff8118af83&gt;] sys_renameat+0x113/0x260 [&lt;ffffffff8118b0eb&gt;] sys_rename+0x1b/0x20 [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> # uanme -r<br/> 2.6.32-279.5.1.el6.x86_64<br/> # cat /proc/fs/lustre/version <br/> lustre: 2.2.93<br/> kernel: patchless_client<br/> build: 2.2.93-gbaaf628-PRISTINE-2.6.32-279.5.1.el6.x86_64 LU-1777

open-by-fid: deadlock in lock_rename()

Bug Minor Reopened Unresolved WC Triage John Hammond open-by-fid Tue, 21 Aug 2012 16:30:18 +0000 Thu, 20 Jul 2017 03:29:53 +0000 Lustre 2.4.0 0 6 <p>Niu</p> <p>Could you please look at this one? It is similar to the work you just did for LU1518</p> <p>Thanks</p> <p>Peter</p> <p>this should be fixed along with <a href="https://jira.whamcloud.com/browse/LU-1518" title="Missing/bad operations in mdd_{obf,dot_lustre}_obj_op causing LBUGs" class="issue-link" data-issue-key="LU-1518"><del>LU-1518</del></a>.</p> <p>ok then let's close this ticket as a duplicate and just ensure that the LU1518 fix cover this case also</p> <p>As per John this was not fixed by the <a href="https://jira.whamcloud.com/browse/LU-1518" title="Missing/bad operations in mdd_{obf,dot_lustre}_obj_op causing LBUGs" class="issue-link" data-issue-key="LU-1518"><del>LU-1518</del></a> after all so reopening</p> <p>I think there isn't a quick fix for such deadlock. We need some way on server side to detect the recursive rename, which should check the 'fid' directory as well.</p> <p>Given that rename files in the 'fid' directory isn't an legal usage, I suggest let's lower the priority of this ticket and fix it in later version.</p> <p>While it would be good to get this fixed for 2.3, since this only affects the client and not the MDS, I'm removing this as a blocker for 2.3 and moving it to 2.4. This isn't a problem that can be hit accidentally. </p> <p>The client is dealock in lock_rename():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">struct dentry *lock_rename(struct dentry *p1, struct dentry *p2) { struct dentry *p; <span class="code-keyword">if</span> (p1 == p2) { mutex_lock_nested(&amp;p1-&gt;d_inode-&gt;i_mutex, I_MUTEX_PARENT); <span class="code-keyword">return</span> NULL; } mutex_lock(&amp;p1-&gt;d_inode-&gt;i_sb-&gt;s_vfs_rename_mutex); p = d_ancestor(p2, p1); <span class="code-keyword">if</span> (p) { mutex_lock_nested(&amp;p2-&gt;d_inode-&gt;i_mutex, I_MUTEX_PARENT); mutex_lock_nested(&amp;p1-&gt;d_inode-&gt;i_mutex, I_MUTEX_CHILD); <span class="code-keyword">return</span> p; } p = d_ancestor(p1, p2); <span class="code-keyword">if</span> (p) { mutex_lock_nested(&amp;p1-&gt;d_inode-&gt;i_mutex, I_MUTEX_PARENT); mutex_lock_nested(&amp;p2-&gt;d_inode-&gt;i_mutex, I_MUTEX_CHILD); <span class="code-keyword">return</span> p; } mutex_lock_nested(&amp;p1-&gt;d_inode-&gt;i_mutex, I_MUTEX_PARENT); mutex_lock_nested(&amp;p2-&gt;d_inode-&gt;i_mutex, I_MUTEX_CHILD); <span class="code-keyword">return</span> NULL; } </pre> </div></div> <p>The root cause is that with 'fid' directory, we can have two directory dentries pointing to the same inode on client, so lock_rename() will try to lock the same inode from two different dentries twice. Without patching kernel, I'm not sure if there is any good way to solve it.</p> <p>Anyway, I don't think it should be a blocker for 2.4. Andreas, any comments? Thanks.</p> <p>Is it possible to block renames that involve the .lustre directory?</p> <p>No, I don't think so. It block renames that involve the 'fid' directory.</p> <p>@Niu Yawei,<br/> I am facing the deadlock in lock_rename while running sanity/test_154a, open_by_fid test.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jul 31 16:10:01 localhost kernel: mrename D 0000000000000000 0 19394 19337 0x00000080 Jul 31 16:10:01 localhost kernel: ffff8800054f1cf8 0000000000000082 0000000000000000 ffffffff811850f5 Jul 31 16:10:01 localhost kernel: 0000000000000000 ffffea00001267b8 ffffffff8100bc0e ffff8800054f1cf8 Jul 31 16:10:01 localhost kernel: ffff88000dffc678 ffff8800054f1fd8 000000000000f4e8 ffff88000dffc678 Jul 31 16:10:01 localhost kernel: Call Trace: Jul 31 16:10:01 localhost kernel: [&lt;ffffffff811850f5&gt;] ? __link_path_walk+0x155/0x1030 Jul 31 16:10:01 localhost kernel: [&lt;ffffffff8100bc0e&gt;] ? apic_timer_interrupt+0xe/0x20 Jul 31 16:10:01 localhost kernel: [&lt;ffffffff8104d92d&gt;] ? mutex_spin_on_owner+0x8d/0xc0 Jul 31 16:10:01 localhost kernel: [&lt;ffffffff814eebbe&gt;] __mutex_lock_slowpath+0x13e/0x180 Jul 31 16:10:01 localhost kernel: [&lt;ffffffff81183b01&gt;] ? path_put+0x31/0x40 Jul 31 16:10:01 localhost kernel: [&lt;ffffffff814eea5b&gt;] mutex_lock+0x2b/0x50 Jul 31 16:10:01 localhost kernel: [&lt;ffffffff81182f83&gt;] lock_rename+0x73/0xe0 Jul 31 16:10:01 localhost kernel: [&lt;ffffffff81186673&gt;] sys_renameat+0x113/0x260 Jul 31 16:10:01 localhost kernel: [&lt;ffffffff81195b70&gt;] ? mntput_no_expire+0x30/0x110 Jul 31 16:10:01 localhost kernel: [&lt;ffffffff81178271&gt;] ? __fput+0x1a1/0x210 Jul 31 16:10:01 localhost kernel: [&lt;ffffffff8113f43e&gt;] ? remove_vma+0x6e/0x90 Jul 31 16:10:01 localhost kernel: [&lt;ffffffff810d4932&gt;] ? audit_syscall_entry+0x272/0x2a0 Jul 31 16:10:01 localhost kernel: [&lt;ffffffff814f2fce&gt;] ? do_page_fault+0x3e/0xa0 Jul 31 16:10:01 localhost kernel: [&lt;ffffffff811867db&gt;] sys_rename+0x1b/0x20 Jul 31 16:10:01 localhost kernel: [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>This is reproducible every time I run sanity/154a. This is lustre 2.1.5 ( esp with backports of <a href="https://jira.whamcloud.com/browse/LU-4279" title=".lustre and .lustre/fid cannot by accessed by FID" class="issue-link" data-issue-key="LU-4279"><del>LU-4279</del></a>, <a href="https://jira.whamcloud.com/browse/LU-3245" title="FS root OBF regression" class="issue-link" data-issue-key="LU-3245"><del>LU-3245</del></a> )<br/> Can you provide some pointers so that I can work on the fix ?</p> <p>Thanks</p> <p>Hello Andreas, Niu Yawei, </p> <p>I have also faced this dead lock while renaming .lustre to .lustre using its fid. i.e</p> <p>echo "rename .lustre to itself"<br/> fid=$($LFS path2fid $DIR)<br/> mrename $DIR/.lustre $DIR/.lustre/fid/$fid/.lustre &amp;&amp;<br/> error "rename .lustre to itself should fail."</p> <p>call trace.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. mrename D 0000000000000000 0 25974 25917 0x00000080 ffff880014401cf8 0000000000000086 ffff880014401d08 ffffffff811850f5 0000000000000000 ffffea00004fe840 ffffffff8100bc0e ffff880014401cf8 ffff8800054afa78 ffff880014401fd8 000000000000f4e8 ffff8800054afa78 Call Trace: [&lt;ffffffff811850f5&gt;] ? __link_path_walk+0x155/0x1030 [&lt;ffffffff8100bc0e&gt;] ? apic_timer_interrupt+0xe/0x20 [&lt;ffffffff8104d92d&gt;] ? mutex_spin_on_owner+0x8d/0xc0 [&lt;ffffffff814eebbe&gt;] __mutex_lock_slowpath+0x13e/0x180 [&lt;ffffffff81183b01&gt;] ? path_put+0x31/0x40 [&lt;ffffffff814eea5b&gt;] mutex_lock+0x2b/0x50 [&lt;ffffffff81182f83&gt;] lock_rename+0x73/0xe0 [&lt;ffffffff81186673&gt;] sys_renameat+0x113/0x260 [&lt;ffffffff81195b70&gt;] ? mntput_no_expire+0x30/0x110 [&lt;ffffffff81178271&gt;] ? __fput+0x1a1/0x210 [&lt;ffffffff8113f43e&gt;] ? remove_vma+0x6e/0x90 [&lt;ffffffff810d4932&gt;] ? audit_syscall_entry+0x272/0x2a0 [&lt;ffffffff814f2fce&gt;] ? do_page_fault+0x3e/0xa0 [&lt;ffffffff811867db&gt;] sys_rename+0x1b/0x20 [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>I have tried to catch this issue in llite layer and return -EPERM from there but not successful. Is this case not currently not supported by lustre or Am i doing something wrong here ?</p> Related LU-1518 Development Rank 1|hzw2cv: Rank (Obsolete) 10464 Severity