<!-- 
RSS generated by JIRA (9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c) at Sat Feb 10 01:45:18 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>Whamcloud Community JIRA</title>
    <link>https://jira.whamcloud.com</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.4.14</version>
        <build-number>940014</build-number>
        <build-date>05-12-2023</build-date>
    </build-info>


<item>
            <title>[LU-4725] wrong lock ordering in rename leads to deadlocks</title>
                <link>https://jira.whamcloud.com/browse/LU-4725</link>
                <project id="10000" key="LU">Lustre</project>
                    <description>&lt;p&gt;the current rename code locks objects in the order: src parent, dst parent, src child, dst child. it may happen that dst is a parent of src, what may lead to deadlock.&lt;/p&gt;

&lt;p&gt;example from a core dump:&lt;br/&gt;
res1 - dst parent&lt;br/&gt;
res2 - dst parent, PDO&lt;br/&gt;
res3 - src parent&lt;/p&gt;

&lt;p&gt;Thread 1 (T1), rename:&lt;br/&gt;
Has RES3 (CW,0x2)&lt;br/&gt;
Wants RES1 (CW,0x2)&lt;/p&gt;

&lt;p&gt;Thread 2 (T2), getattr:&lt;br/&gt;
Has RES1(CR,0x2)&lt;br/&gt;
Has RES2(PR,0x2)&lt;br/&gt;
Wants RES3(PR,0x2) - blocked by T1&lt;/p&gt;

&lt;p&gt;Thread 3 (T3), create or open|create&lt;br/&gt;
Has RES1(CW,0x2)&lt;br/&gt;
Wants RES2(PW,0x2) - blocked by T2&lt;/p&gt;

&lt;p&gt;Thread4 (T4), getattr or similar&lt;br/&gt;
Wants RES1(PR,0x2) - blocked by T3&lt;/p&gt;

&lt;p&gt;T1 has no conflicts, but is sitting in the waiting queue behind T4, thus not granted.&lt;/p&gt;</description>
                <environment></environment>
        <key id="23498">LU-4725</key>
            <summary>wrong lock ordering in rename leads to deadlocks</summary>
                <type id="1" iconUrl="https://jira.whamcloud.com/secure/viewavatar?size=xsmall&amp;avatarId=11303&amp;avatarType=issuetype">Bug</type>
                                            <priority id="1" iconUrl="https://jira.whamcloud.com/images/icons/priorities/blocker.svg">Blocker</priority>
                        <status id="5" iconUrl="https://jira.whamcloud.com/images/icons/statuses/resolved.png" description="A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.">Resolved</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="1">Fixed</resolution>
                                        <assignee username="hongchao.zhang">Hongchao Zhang</assignee>
                                    <reporter username="vitaly_fertman">Vitaly Fertman</reporter>
                        <labels>
                            <label>patch</label>
                    </labels>
                <created>Thu, 6 Mar 2014 21:00:12 +0000</created>
                <updated>Wed, 8 Mar 2023 01:45:22 +0000</updated>
                            <resolved>Mon, 16 Jun 2014 14:28:20 +0000</resolved>
                                    <version>Lustre 2.1.6</version>
                    <version>Lustre 2.6.0</version>
                    <version>Lustre 2.5.2</version>
                    <version>Lustre 2.4.3</version>
                                    <fixVersion>Lustre 2.6.0</fixVersion>
                    <fixVersion>Lustre 2.5.4</fixVersion>
                                        <due></due>
                            <votes>0</votes>
                                    <watches>15</watches>
                                                                            <comments>
                            <comment id="78645" author="vitaly_fertman" created="Thu, 6 Mar 2014 22:40:46 +0000"  >&lt;p&gt;CODE: &lt;a href=&quot;http://review.whamcloud.com/9538&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/9538&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="79001" author="adilger" created="Tue, 11 Mar 2014 15:03:41 +0000"  >&lt;p&gt;Vitaly, does this problem also affect 2.5 or 2.4?&lt;/p&gt;</comment>
                            <comment id="79149" author="jlevi" created="Wed, 12 Mar 2014 17:09:20 +0000"  >&lt;p&gt;HongChao, &lt;br/&gt;
Could you please keep an eye on this patch?&lt;/p&gt;</comment>
                            <comment id="79365" author="paf" created="Fri, 14 Mar 2014 19:35:07 +0000"  >&lt;p&gt;Andreas - This problem report&apos;s originally from Cray.  I can confirm it seems to affect all current versions of Lustre.  At least, we&apos;ve seen it in 2.1 (Xyratex&apos;s server code), and Cray&apos;s 2.4 and 2.5 servers.&lt;/p&gt;

&lt;p&gt;And looking at the proposed cause (the rename locking order that comes up when moving an object up in to its grandparent), that&apos;s the same in all of those versions, so it makes sense.&lt;/p&gt;</comment>
                            <comment id="79503" author="vitaly_fertman" created="Mon, 17 Mar 2014 12:29:06 +0000"  >&lt;p&gt;Andreas, this ordering has not changed since was introduced in 2.0.&lt;br/&gt;
(sorry for the delay, I was on PTO)&lt;/p&gt;</comment>
                            <comment id="81863" author="jlevi" created="Thu, 17 Apr 2014 18:14:33 +0000"  >&lt;p&gt;HongChao,&lt;br/&gt;
Can you please refresh this patch?&lt;br/&gt;
Thank you!&lt;/p&gt;</comment>
                            <comment id="81938" author="hongchao.zhang" created="Fri, 18 Apr 2014 13:34:06 +0000"  >&lt;p&gt;the patch has been updated&lt;/p&gt;</comment>
                            <comment id="82042" author="jlevi" created="Mon, 21 Apr 2014 12:49:52 +0000"  >&lt;p&gt;Test was retriggered today.&lt;/p&gt;</comment>
                            <comment id="83625" author="pjones" created="Fri, 9 May 2014 14:50:35 +0000"  >&lt;p&gt;Landed for 2.6&lt;/p&gt;</comment>
                            <comment id="84270" author="adilger" created="Fri, 16 May 2014 17:36:49 +0000"  >&lt;p&gt;Reopened because of a new problem found.  &lt;a href=&quot;http://review.whamcloud.com/10342&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/10342&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="84452" author="green" created="Tue, 20 May 2014 15:21:36 +0000"  >&lt;p&gt;Sothe current deadlocks in rename I am observinghave a trace like this:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;[24234.786010] Call Trace:
[24234.786802]  [&amp;lt;ffffffffa052a304&amp;gt;] ? htable_lookup+0x1c4/0x1e0 [obdclass]
[24234.787351]  [&amp;lt;ffffffffa052a91b&amp;gt;] lu_object_find_at+0xab/0x350 [obdclass]
[24234.787869]  [&amp;lt;ffffffff8105de00&amp;gt;] ? default_wake_function+0x0/0x20
[24234.788398]  [&amp;lt;ffffffffa052abd6&amp;gt;] lu_object_find+0x16/0x20 [obdclass]
[24234.788918]  [&amp;lt;ffffffffa0ce4e06&amp;gt;] mdt_object_find+0x56/0x170 [mdt]
[24234.789480]  [&amp;lt;ffffffffa0d03e32&amp;gt;] mdt_rename_sanity+0xa2/0x4a0 [mdt]
[24234.790006]  [&amp;lt;ffffffffa0d08ad8&amp;gt;] mdt_reint_rename_internal+0xe08/0x1a80 [mdt]
[24234.790890]  [&amp;lt;ffffffffa06be630&amp;gt;] ? ldlm_lock_enqueue+0x1d0/0x8b0 [ptlrpc]
[24234.791424]  [&amp;lt;ffffffffa0d09994&amp;gt;] mdt_reint_rename_or_migrate+0x244/0x660 [mdt]
[24234.792291]  [&amp;lt;ffffffffa06dc6c0&amp;gt;] ? ldlm_blocking_ast+0x0/0x180 [ptlrpc]
[24234.792832]  [&amp;lt;ffffffffa06ddd30&amp;gt;] ? ldlm_completion_ast+0x0/0x930 [ptlrpc]
[24234.793414]  [&amp;lt;ffffffffa0d09de0&amp;gt;] mdt_reint_rename+0x10/0x20 [mdt]
[24234.793959]  [&amp;lt;ffffffffa0d024b1&amp;gt;] mdt_reint_rec+0x41/0xe0 [mdt]
[24234.795042]  [&amp;lt;ffffffffa0ce7e03&amp;gt;] mdt_reint_internal+0x4c3/0x7c0 [mdt]
[24234.795374]  [&amp;lt;ffffffffa0ce868b&amp;gt;] mdt_reint+0x6b/0x120 [mdt]
[24234.795915]  [&amp;lt;ffffffffa0767b7c&amp;gt;] tgt_request_handle+0x23c/0xac0 [ptlrpc]
[24234.796475]  [&amp;lt;ffffffffa0718fb8&amp;gt;] ptlrpc_main+0xcc8/0x1950 [ptlrpc]
[24234.797078]  [&amp;lt;ffffffffa07182f0&amp;gt;] ? ptlrpc_main+0x0/0x1950 [ptlrpc]
[24234.797579]  [&amp;lt;ffffffff81098c06&amp;gt;] kthread+0x96/0xa0
[24234.798041]  [&amp;lt;ffffffff8100c24a&amp;gt;] child_rip+0xa/0x20
[24234.798495]  [&amp;lt;ffffffff81098b70&amp;gt;] ? kthread+0x0/0xa0
[24234.798981]  [&amp;lt;ffffffff8100c240&amp;gt;] ? child_rip+0x0/0x20
[24234.799449] 
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;(gdb) l *mdt_rename_sanity+0xa2
0x23e82 is in mdt_rename_sanity (/home/green/git/lustre-release-double/lustre/mdt/mdt_reint.c:1263).
1258			return 0;
1259	
1260		do {
1261			LASSERT(fid_is_sane(&amp;amp;dst_fid));
1262			dst = mdt_object_find(info-&amp;gt;mti_env, info-&amp;gt;mti_mdt, &amp;amp;dst_fid);
1263			if (!IS_ERR(dst)) {
1264				rc = mdo_is_subdir(info-&amp;gt;mti_env,
1265						   mdt_object_child(dst), fid,
1266						   &amp;amp;dst_fid);
1267				mdt_object_put(info-&amp;gt;mti_env, dst);
(gdb) l *(mdt_reint_rename_internal+0xe08)
0x28b28 is in mdt_reint_rename_internal (/home/green/git/lustre-release-double/lustre/mdt/mdt_reint.c:1783).
1778			GOTO(out_unlock_parents, rc = PTR_ERR(mold));
1779	
1780		/* Check if @mtgtdir is subdir of @mold, before locking child
1781		 * to avoid reverse locking. */
1782		rc = mdt_rename_sanity(info, rr-&amp;gt;rr_fid2, old_fid);
1783		if (rc)
1784			GOTO(out_put_old, rc);
1785	
1786		tgt_vbr_obj_set(info-&amp;gt;mti_env, mdt_obj2dt(mold));
1787		/* save version after locking */
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;The apparent problem is such that while we are holding object reference on target (or it&apos;s parent) after we looked it up it got unlinked and attempted to be destroyed, but we are trying to find it again ewhile lholding one reference and get deadlocked in find_object_at waiting for all referenced to go away due to HEARD_BANSHEE being set.&lt;/p&gt;</comment>
                            <comment id="85111" author="vitaly_fertman" created="Thu, 29 May 2014 10:43:26 +0000"  >&lt;p&gt;alternatives for the last problem:&lt;br/&gt;
&lt;a href=&quot;http://review.whamcloud.com/10342&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/10342&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;http://review.whamcloud.com/10484&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/10484&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="85556" author="vitaly_fertman" created="Tue, 3 Jun 2014 11:11:51 +0000"  >&lt;p&gt;independently of the previous fixes, but still related to rename, another deadlock is found:&lt;br/&gt;
&lt;a href=&quot;http://review.whamcloud.com/10570&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/10570&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="85742" author="vitaly_fertman" created="Wed, 4 Jun 2014 19:35:22 +0000"  >&lt;p&gt;the last patch is moved to a separate ticket: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-5144&quot; title=&quot;rename vs link deadlock&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-5144&quot;&gt;&lt;del&gt;LU-5144&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="86673" author="jlevi" created="Mon, 16 Jun 2014 14:28:20 +0000"  >&lt;p&gt;Patches landed to Master. Closing this ticket and &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-5144&quot; title=&quot;rename vs link deadlock&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-5144&quot;&gt;&lt;del&gt;LU-5144&lt;/del&gt;&lt;/a&gt; as the patch moved to that ticket was also landed. &lt;/p&gt;</comment>
                            <comment id="87871" author="rdeshmukh_xyratex" created="Tue, 1 Jul 2014 10:54:41 +0000"  >&lt;p&gt;I have back ported &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-4725&quot; title=&quot;wrong lock ordering in rename leads to deadlocks&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-4725&quot;&gt;&lt;del&gt;LU-4725&lt;/del&gt;&lt;/a&gt; patches to b2_5 and here are the links:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;http://review.whamcloud.com/#/c/10916/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/#/c/10916/&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;http://review.whamcloud.com/#/c/10917/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/#/c/10917/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Not sure if I need to create new bug for this? Please help.&lt;/p&gt;</comment>
                            <comment id="90678" author="rdeshmukh_xyratex" created="Mon, 4 Aug 2014 07:34:12 +0000"  >&lt;p&gt;The result for first patch shows two tests, this because first patch is not complete.&lt;br/&gt;
All 19 tests pass for second patch. (I was not sure where to mention it in gerrit, so updating here).&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;http://review.whamcloud.com/#/c/10916/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/#/c/10916/&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;http://review.whamcloud.com/#/c/10917/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/#/c/10917/&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="92577" author="yujian" created="Wed, 27 Aug 2014 05:48:38 +0000"  >&lt;p&gt;Just combined the above two patches into one: &lt;a href=&quot;http://review.whamcloud.com/11615&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/11615&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="25115">LU-5178</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10011">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="26081">LU-5514</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="24838">LU-5111</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="74811">LU-16589</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="25115">LU-5178</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                            <customfield id="customfield_10890" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10390" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzwgy7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10090" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>12984</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10060" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Severity</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10022"><![CDATA[3]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        </customfields>
    </item>
</channel>
</rss>