<!-- 
RSS generated by JIRA (9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c) at Sat Feb 10 02:20:19 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>Whamcloud Community JIRA</title>
    <link>https://jira.whamcloud.com</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.4.14</version>
        <build-number>940014</build-number>
        <build-date>05-12-2023</build-date>
    </build-info>


<item>
            <title>[LU-8760] sanity-lfsck test 31g hung</title>
                <link>https://jira.whamcloud.com/browse/LU-8760</link>
                <project id="10000" key="LU">Lustre</project>
                    <description>&lt;p&gt;With patch &lt;a href=&quot;http://review.whamcloud.com/7200&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/7200&lt;/a&gt; on master branch, sanity-lfsck test 31g hung as follows:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;== sanity-lfsck test 31g: Repair the corrupted slave LMV EA ========================================== 06:38:18 (1476970698)
#####
For some reason, the stripe index in the slave LMV EA is
corrupted. The LFSCK should repair the slave LMV EA.
#####
Inject failure stub on MDT0 to simulate the case that the
slave LMV EA on the first shard of the striped directory
claims the same index as the second shard claims
CMD: onyx-35vm7 /usr/sbin/lctl set_param fail_loc=0x162b fail_val=0
fail_loc=0x162b
fail_val=0
CMD: onyx-35vm7 /usr/sbin/lctl set_param fail_loc=0x0 fail_val=0
fail_loc=0x0
fail_val=0
Trigger namespace LFSCK to repair the slave LMV EA
CMD: onyx-35vm7 /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A
Started LFSCK on the device lustre-MDT0000: scrub namespace
CMD: onyx-35vm7 /usr/sbin/lctl lfsck_query -t namespace -M lustre-MDT0000 -w |
		      awk &apos;/^namespace_mdts_completed/ { print \$2 }&apos;
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Maloo report: &lt;a href=&quot;https://testing.hpdd.intel.com/test_sets/7e295098-96fa-11e6-a763-5254006e85c2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://testing.hpdd.intel.com/test_sets/7e295098-96fa-11e6-a763-5254006e85c2&lt;/a&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="41056">LU-8760</key>
            <summary>sanity-lfsck test 31g hung</summary>
                <type id="1" iconUrl="https://jira.whamcloud.com/secure/viewavatar?size=xsmall&amp;avatarId=11303&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.whamcloud.com/images/icons/priorities/major.svg">Major</priority>
                        <status id="5" iconUrl="https://jira.whamcloud.com/images/icons/statuses/resolved.png" description="A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.">Resolved</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="1">Fixed</resolution>
                                        <assignee username="yong.fan">nasf</assignee>
                                    <reporter username="yujian">Jian Yu</reporter>
                        <labels>
                    </labels>
                <created>Wed, 26 Oct 2016 14:20:16 +0000</created>
                <updated>Wed, 24 Aug 2022 08:21:24 +0000</updated>
                            <resolved>Mon, 8 Apr 2019 14:20:48 +0000</resolved>
                                    <version>Lustre 2.9.0</version>
                                    <fixVersion>Lustre 2.10.1</fixVersion>
                    <fixVersion>Lustre 2.11.0</fixVersion>
                    <fixVersion>Lustre 2.13.0</fixVersion>
                                        <due></due>
                            <votes>0</votes>
                                    <watches>8</watches>
                                                                            <comments>
                            <comment id="172172" author="yong.fan" created="Thu, 3 Nov 2016 14:49:57 +0000"  >&lt;p&gt;The failure itself is not important. But the reason the failure may be well affect the whole Lustre.&lt;br/&gt;
The logs show that the namespace LFSCK on the 2nd MDS deadlock as following:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;07:39:35:[21041.906022] lfsck           S ffff880053dff300     0 20397      2 0x00000080
07:39:35:[21041.906022]  ffff880053fa7c90 0000000000000046 ffff880053dff300 ffff880053fa7fd8
07:39:35:[21041.906022]  ffff880053fa7fd8 ffff880053fa7fd8 ffff880053dff300 ffff8800457ecc00
07:39:35:[21041.906022]  ffff880077f06800 ffff88004e3a80d0 0000000000000000 ffff880053dff300
07:39:35:[21041.906022] Call Trace:
07:39:35:[21041.906022]  [&amp;lt;ffffffff8163bc39&amp;gt;] schedule+0x29/0x70
07:39:35:[21041.906022]  [&amp;lt;ffffffffa0ca596e&amp;gt;] lfsck_double_scan_generic+0x22e/0x2c0 [lfsck]
07:39:35:[21041.906022]  [&amp;lt;ffffffff810b8940&amp;gt;] ? wake_up_state+0x20/0x20
07:39:35:[21041.906022]  [&amp;lt;ffffffffa0caed90&amp;gt;] lfsck_namespace_double_scan+0x30/0x140 [lfsck]
07:39:35:[21041.906022]  [&amp;lt;ffffffffa0ca5cc9&amp;gt;] lfsck_double_scan+0x59/0x200 [lfsck]
07:39:35:[21041.906022]  [&amp;lt;ffffffffa0c4961a&amp;gt;] ? osd_otable_it_fini+0xca/0x240 [osd_ldiskfs]
07:39:35:[21041.906022]  [&amp;lt;ffffffff811c0bed&amp;gt;] ? kfree+0xfd/0x140
07:39:35:[21041.906022]  [&amp;lt;ffffffffa0caab04&amp;gt;] lfsck_master_engine+0x434/0x1310 [lfsck]
07:39:35:[21041.906022]  [&amp;lt;ffffffff810b8940&amp;gt;] ? wake_up_state+0x20/0x20
07:39:35:[21041.906022]  [&amp;lt;ffffffffa0caa6d0&amp;gt;] ? lfsck_master_oit_engine+0x14c0/0x14c0 [lfsck]
07:39:35:[21041.906022]  [&amp;lt;ffffffff810a5b8f&amp;gt;] kthread+0xcf/0xe0
07:39:35:[21041.906022]  [&amp;lt;ffffffff810a5ac0&amp;gt;] ? kthread_create_on_node+0x140/0x140
07:39:35:[21041.906022]  [&amp;lt;ffffffff81646b98&amp;gt;] ret_from_fork+0x58/0x90
07:39:35:[21041.906022]  [&amp;lt;ffffffff810a5ac0&amp;gt;] ? kthread_create_on_node+0x140/0x140
07:39:35:[21041.906022] lfsck_namespace S ffffffffa0d0fba0     0 20399      2 0x00000080
07:39:35:[21041.906022]  ffff8800793f7d48 0000000000000046 ffff880053df8b80 ffff8800793f7fd8
07:39:35:[21041.906022]  ffff8800793f7fd8 ffff8800793f7fd8 ffff880053df8b80 0000000000000000
07:39:35:[21041.906022]  ffff8800457ecc08 ffff8800457ecc00 ffff88004e3a8000 ffffffffa0d0fba0
07:39:35:[21041.906022] Call Trace:
07:39:35:[21041.906022]  [&amp;lt;ffffffff8163bc39&amp;gt;] schedule+0x29/0x70
07:39:35:[21041.906022]  [&amp;lt;ffffffffa0cacbdd&amp;gt;] lfsck_assistant_engine+0x11fd/0x2150 [lfsck]
07:39:35:[21041.906022]  [&amp;lt;ffffffff810c1a96&amp;gt;] ? dequeue_entity+0x106/0x520
07:39:35:[21041.906022]  [&amp;lt;ffffffff8163b5e8&amp;gt;] ? __schedule+0x2d8/0x900
07:39:35:[21041.906022]  [&amp;lt;ffffffff810b8940&amp;gt;] ? wake_up_state+0x20/0x20
07:39:35:[21041.906022]  [&amp;lt;ffffffffa0cab9e0&amp;gt;] ? lfsck_master_engine+0x1310/0x1310 [lfsck]
07:39:35:[21041.906022]  [&amp;lt;ffffffff810a5b8f&amp;gt;] kthread+0xcf/0xe0
07:39:35:[21041.906022]  [&amp;lt;ffffffff810a5ac0&amp;gt;] ? kthread_create_on_node+0x140/0x140
07:39:35:[21041.906022]  [&amp;lt;ffffffff81646b98&amp;gt;] ret_from_fork+0x58/0x90
07:39:35:[21041.906022]  [&amp;lt;ffffffff810a5ac0&amp;gt;] ? kthread_create_on_node+0x140/0x140
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;Debug according to the stack:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;# gdb lfsck.ko
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-64.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later &amp;lt;http://gnu.org/licenses/gpl.html&amp;gt;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type &quot;show copying&quot;
and &quot;show warranty&quot; for details.
This GDB was configured as &quot;x86_64-redhat-linux-gnu&quot;.
For bug reporting instructions, please see:
&amp;lt;http://www.gnu.org/software/gdb/bugs/&amp;gt;...
Reading symbols from /root/Work/Lustre/master/lustre-release/lustre/lfsck/lfsck.ko...done.
(gdb) l *lfsck_double_scan_generic+0x22e
0x1197e is in lfsck_double_scan_generic (/root/Work/Lustre/master/lustre-release/lustre/lfsck/lfsck_lib.c:2606).
2601		CDEBUG(D_LFSCK, &quot;%s: waiting for assistant to do %s double_scan, &quot;
2602		       &quot;status %d\n&quot;,
2603		       lfsck_lfsck2name(com-&amp;gt;lc_lfsck), lad-&amp;gt;lad_name, status);
2604	
2605		wake_up_all(&amp;amp;athread-&amp;gt;t_ctl_waitq);
2606		l_wait_event(mthread-&amp;gt;t_ctl_waitq,
2607			     lad-&amp;gt;lad_in_double_scan ||
2608			     thread_is_stopped(athread),
2609			     &amp;amp;lwi);
2610	
(gdb) l *lfsck_assistant_engine+0x11fd
0x18bfd is in lfsck_assistant_engine (/root/Work/Lustre/master/lustre-release/lustre/lfsck/lfsck_engine.c:1628).
warning: Source file is more recent than executable.
1623				lao-&amp;gt;la_req_fini(env, lar);
1624				if (rc &amp;lt; 0 &amp;amp;&amp;amp; bk-&amp;gt;lb_param &amp;amp; LPF_FAILOUT)
1625					GOTO(cleanup, rc);
1626			}
1627	
1628			l_wait_event(athread-&amp;gt;t_ctl_waitq,
1629				     !lfsck_assistant_req_empty(lad) ||
1630				     lad-&amp;gt;lad_exit ||
1631				     lad-&amp;gt;lad_to_post ||
1632				     lad-&amp;gt;lad_to_double_scan,
(gdb) 
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;That means the master engine &quot;lfsck&quot; was blocked in the function lfsck_double_scan_generic(), and wait the assistant thread &quot;lfsck_namespace&quot; to complete  the second-stage scanning:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;int lfsck_double_scan_generic(const struct lu_env *env,
                              struct lfsck_component *com, int status)
{
        struct lfsck_assistant_data     *lad     = com-&amp;gt;lc_data;
        struct ptlrpc_thread            *mthread = &amp;amp;com-&amp;gt;lc_lfsck-&amp;gt;li_thread;
        struct ptlrpc_thread            *athread = &amp;amp;lad-&amp;gt;lad_thread;
        struct l_wait_info               lwi     = { 0 };

        if (status != LS_SCANNING_PHASE2)
                lad-&amp;gt;lad_exit = 1;
        else
                lad-&amp;gt;lad_to_double_scan = 1;

        CDEBUG(D_LFSCK, &quot;%s: waiting for assistant to do %s double_scan, &quot;
               &quot;status %d\n&quot;,
               lfsck_lfsck2name(com-&amp;gt;lc_lfsck), lad-&amp;gt;lad_name, status);

        wake_up_all(&amp;amp;athread-&amp;gt;t_ctl_waitq);
(line 2606)==&amp;gt;        l_wait_event(mthread-&amp;gt;t_ctl_waitq,
                     lad-&amp;gt;lad_in_double_scan ||
                     thread_is_stopped(athread),
                     &amp;amp;lwi);

        CDEBUG(D_LFSCK, &quot;%s: the assistant has done %s double_scan, &quot;
               &quot;status %d\n&quot;, lfsck_lfsck2name(com-&amp;gt;lc_lfsck), lad-&amp;gt;lad_name,
               lad-&amp;gt;lad_assistant_status);

        if (lad-&amp;gt;lad_assistant_status &amp;lt; 0)
                return lad-&amp;gt;lad_assistant_status;

        return 0;
}
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;While the assistant engine &quot;lfsck_namespace&quot; was waiting for the master engine to unplug the flag &quot;lad_to_double_scan&quot; or &quot;lad-&amp;gt;lad_exit&quot; as following:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;int lfsck_assistant_engine(void *args)
{
...
(line 1628)==&amp;gt;                l_wait_event(athread-&amp;gt;t_ctl_waitq,
                             !lfsck_assistant_req_empty(lad) ||
                             lad-&amp;gt;lad_exit ||
                             lad-&amp;gt;lad_to_post ||
                             lad-&amp;gt;lad_to_double_scan,
                             &amp;amp;lwi);
...
}
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;In fact, the &quot;lfsck&quot; thread has already set the &quot;lad-&amp;gt;lad_to_double_scan&quot; or &quot;lad-&amp;gt;lad_exit&quot; before waking up the &quot;lfsck_namespace&quot; thread. Then the &quot;lfsck&quot; thread went to wait. But the &quot;lfsck_namespace&quot; thread did NOT found the condition. The logic is not complex, the issue should be inside the l_wait_event() that is widely used for kinds of Lustre functions.&lt;/p&gt;

&lt;p&gt;I suspect that it is related with some race conditions because of out-of-order execution in the following code:&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;#define __l_wait_event(wq, condition, info, ret, l_add_wait)                   \
do {                                                                           \
        wait_queue_t __wait;                                                   \
...
        for (;;) {                                                             \
(line 278)==&amp;gt;                set_current_state(TASK_INTERRUPTIBLE);                         \
                                                                               \
(line 280)==&amp;gt;                if (condition)                                                 \
                        break;                                                 \

                if (__timeout == 0) {                                          \
                        schedule();                                            \
...
}
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;

&lt;p&gt;For modern CPU, there may be out-of-oder execution between changing thread&apos;s state (line 278) and checking conditions (line 280). Consider the following real execution order:&lt;/p&gt;

&lt;p&gt;1. Thread1 checks condition on CPU1, gets false.&lt;br/&gt;
2. Thread2 sets condition on CPU2.&lt;br/&gt;
3. Thread2 calls wake_up() on CPU2 to wake the threads with state TASK_INTERRUPTIBLE | TASK_UNINTERRUPTIBLE. But the Thread1&apos;s state is TASK_RUNNING at that time.&lt;br/&gt;
4. Thread1 sets its state as TASK_INTERRUPTIBLE on CPU1, then schedule.&lt;/p&gt;

&lt;p&gt;If the &apos;__timeout&apos; variable is zero, the Thread1 will have no chance to check the condition again. Generally, the interval between out-of-ordered step1 and step4 is very tiny, as to above step2 and step3 cannot happen. On some degree, it can explain why we seldom hit related trouble. But such race really exists, especially consider that the step1 and step4 can be interruptible.&lt;/p&gt;

&lt;p&gt;So I will make patch to add smp_mb() between changing thread&apos;s state and checking condition to avoid out-of-order execution.&lt;/p&gt;</comment>
                            <comment id="172175" author="gerrit" created="Thu, 3 Nov 2016 15:05:55 +0000"  >&lt;p&gt;Fan Yong (fan.yong@intel.com) uploaded a new patch: &lt;a href=&quot;http://review.whamcloud.com/23564&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://review.whamcloud.com/23564&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-8760&quot; title=&quot;sanity-lfsck test 31g hung&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-8760&quot;&gt;&lt;del&gt;LU-8760&lt;/del&gt;&lt;/a&gt; lib: avoid unexpected out of order execution&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: a66b0c9809dd4e0ce75944a4763e333421bae8fd&lt;/p&gt;</comment>
                            <comment id="203846" author="gerrit" created="Sat, 29 Jul 2017 00:02:16 +0000"  >&lt;p&gt;Oleg Drokin (oleg.drokin@intel.com) merged in patch &lt;a href=&quot;https://review.whamcloud.com/23564/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/23564/&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-8760&quot; title=&quot;sanity-lfsck test 31g hung&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-8760&quot;&gt;&lt;del&gt;LU-8760&lt;/del&gt;&lt;/a&gt; lib: avoid unexpected out of order execution&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: &lt;br/&gt;
Commit: c2b6030e9217e54e7153c0a33cce0c2ea4afa54c&lt;/p&gt;</comment>
                            <comment id="204216" author="mdiep" created="Wed, 2 Aug 2017 16:10:36 +0000"  >&lt;p&gt;Landed for 2.11&lt;/p&gt;</comment>
                            <comment id="204217" author="gerrit" created="Wed, 2 Aug 2017 16:11:37 +0000"  >&lt;p&gt;Minh Diep (minh.diep@intel.com) uploaded a new patch: &lt;a href=&quot;https://review.whamcloud.com/28322&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/28322&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-8760&quot; title=&quot;sanity-lfsck test 31g hung&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-8760&quot;&gt;&lt;del&gt;LU-8760&lt;/del&gt;&lt;/a&gt; lib: avoid unexpected out of order execution&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: b2_10&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: 11b5fddf2dc3c40cdf9bce8cd19db8f162a5dffb&lt;/p&gt;</comment>
                            <comment id="207648" author="gerrit" created="Wed, 6 Sep 2017 16:31:28 +0000"  >&lt;p&gt;John L. Hammond (john.hammond@intel.com) merged in patch &lt;a href=&quot;https://review.whamcloud.com/28322/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/28322/&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-8760&quot; title=&quot;sanity-lfsck test 31g hung&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-8760&quot;&gt;&lt;del&gt;LU-8760&lt;/del&gt;&lt;/a&gt; lib: avoid unexpected out of order execution&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: b2_10&lt;br/&gt;
Current Patch Set: &lt;br/&gt;
Commit: 9785fb53d0c939b2d94a69a580bdf0b6d968a25e&lt;/p&gt;</comment>
                            <comment id="244659" author="aboyko" created="Tue, 26 Mar 2019 11:39:45 +0000"  >&lt;p&gt;I have the same failure for 2.12 version.&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;crash&amp;gt; ps | grep lfs
   6537      2   7  ffff880ffb9aeeb0  IN   0.0       0      0  [lfsck]
   6539      2   6  ffff880ffb9aaf70  IN   0.0       0      0  [lfsck_layout]
   6540      2   0  ffff880ffb9acf10  IN   0.0       0      0  [lfsck_namespace]
crash&amp;gt; bt 6537
PID: 6537   TASK: ffff880ffb9aeeb0  CPU: 7   COMMAND: &quot;lfsck&quot;
 #0 [ffff880178d13c08] __schedule at ffffffff816b6de4
 #1 [ffff880178d13c90] schedule at ffffffff816b7409
 #2 [ffff880178d13ca0] lfsck_double_scan_generic at ffffffffc115a66e [lfsck]
 #3 [ffff880178d13d18] lfsck_layout_master_double_scan at ffffffffc1181bc0 [lfsck]
 #4 [ffff880178d13d60] lfsck_double_scan at ffffffffc115af0f [lfsck]
 #5 [ffff880178d13df0] lfsck_master_engine at ffffffffc115fe16 [lfsck]
 #6 [ffff880178d13ec8] kthread at ffffffff810b4031
 #7 [ffff880178d13f50] ret_from_fork at ffffffff816c455d
crash&amp;gt; bt 6540
PID: 6540   TASK: ffff880ffb9acf10  CPU: 0   COMMAND: &quot;lfsck_namespace&quot;
 #0 [ffff880fa73e3ce8] __schedule at ffffffff816b6de4
 #1 [ffff880fa73e3d70] schedule at ffffffff816b7409
 #2 [ffff880fa73e3d80] lfsck_assistant_engine at ffffffffc1161e0d [lfsck]
 #3 [ffff880fa73e3ec8] kthread at ffffffff810b4031
 #4 [ffff880fa73e3f50] ret_from_fork at ffffffff816c455d
crash&amp;gt; bt 6539
PID: 6539   TASK: ffff880ffb9aaf70  CPU: 6   COMMAND: &quot;lfsck_layout&quot;
 #0 [ffff880f67be7ce8] __schedule at ffffffff816b6de4
 #1 [ffff880f67be7d70] schedule at ffffffff816b7409
 #2 [ffff880f67be7d80] lfsck_assistant_engine at ffffffffc1161e0d [lfsck]
 #3 [ffff880f67be7ec8] kthread at ffffffff810b4031
 #4 [ffff880f67be7f50] ret_from_fork at ffffffff816c455d
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;lfsck_master_engine-&amp;gt;lfsck_double_scan_generic() sleeps and waits wakeup from lfsck_layout-&amp;gt;lfsck_assistant_engine(). And lfsck_assistant_engine sleeps and wait when lfsck_master_engine starts some operation.&lt;br/&gt;
 The state of the lfsck_assistant_data is&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;struct lfsck_assistant_data {
  lad_lock = {
    {
      rlock = {
        raw_lock = {
          val = {
            counter = 0
          }
        }
      }
    }
  },
  lad_req_list = {
    next = 0xffff880fa720d208,
    prev = 0xffff880fa720d208
  },
  lad_ost_list = {
    next = 0xffff880fd8680ce8,
    prev = 0xffff880ff9310568
  },
  lad_ost_phase1_list = {
    next = 0xffff880fd8680cf8,
    prev = 0xffff880ff9310578
  },
  lad_ost_phase2_list = {
    next = 0xffff880fa720d238,
    prev = 0xffff880fa720d238
  },
  lad_mdt_list = {
    next = 0xffff880fa720d248,
    prev = 0xffff880fa720d248
  },
  lad_mdt_phase1_list = {
    next = 0xffff880fa720d258,
    prev = 0xffff880fa720d258
  },
  lad_mdt_phase2_list = {
    next = 0xffff880fa720d268,
    prev = 0xffff880fa720d268
  },
  lad_name = 0xffffffffc11b2423 &quot;lfsck_layout&quot;,
  lad_thread = {
    t_link = {
      next = 0x0,
      prev = 0x0
    },
    t_data = 0x0,
    t_flags = 8,
    t_id = 0,
    t_pid = 0,
    t_watchdog = 0x0,
    t_svcpt = 0x0,
    t_ctl_waitq = {
      lock = {
        {
          rlock = {
            raw_lock = {
              val = {
                counter = 0
              }
            }
          }
        }
      },
      task_list = {
        next = 0xffff880f67be7e80,
        prev = 0xffff880f67be7e80
      }
    },
    t_env = 0x0,
    t_name = &quot;\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000&quot;
  },
  lad_task = 0xffff880ffb9aaf70,
  lad_ops = 0xffffffffc11d2d40 &amp;lt;lfsck_layout_assistant_ops&amp;gt;,
  lad_bitmap = 0xffff88103b1862e0,
  lad_touch_gen = 97,
  lad_prefetched = 0,
  lad_assistant_status = 0,
  lad_post_result = 1,
  lad_to_post = 0,
  lad_to_double_scan = 0,
  lad_in_double_scan = 0,
  lad_exit = 0,
  lad_incomplete = 0,
  lad_advance_lock = false
}
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;Before the sleep lfsck_double_scan_generic set lad_to_double_scan to 1. lfsck_assistant_engine could zeroed it and set lad_in_double_scan to 1. But the data doesn&apos;t show this.&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;                if (lad-&amp;gt;lad_to_double_scan) {
                        lad-&amp;gt;lad_to_double_scan = 0;
                        atomic_inc(&amp;amp;lfsck-&amp;gt;li_double_scan_count);
                        lad-&amp;gt;lad_in_double_scan = 1;
                        wake_up_all(&amp;amp;mthread-&amp;gt;t_ctl_waitq);

&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;li_double_scan_count is 0 too.&lt;br/&gt;
 The sleep state is equal to current issue, the first comment shows the same. But the patch already included at 2.12. So the real root cause is different for both fails I think.&lt;br/&gt;
 The log shows setting lad_to_double_scan and clearing lad_to_post at the same time.&lt;/p&gt;
&lt;div class=&quot;preformatted panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;preformattedContent panelContent&quot;&gt;
&lt;pre&gt;00100000:10000000:7.0:1552569516.765338:0:6537:0:(lfsck_namespace.c:4595:lfsck_namespace_post()) testfs-MDT0001-osd: namespace LFSCK post done: rc = 0
00100000:10000000:6.0:1552569516.765352:0:6539:0:(lfsck_engine.c:1661:lfsck_assistant_engine()) testfs-MDT0001-osd: lfsck_layout LFSCK assistant thread post
00100000:10000000:7.0:1552569516.765354:0:6537:0:(lfsck_lib.c:2614:lfsck_double_scan_generic()) testfs-MDT0001-osd: waiting for assistant to do lfsck_layout double_scan, status 2

00100000:10000000:6.0:1552569516.765473:0:6539:0:(lfsck_engine.c:1680:lfsck_assistant_engine()) testfs-MDT0001-osd: LFSCK assistant notified others for lfsck_layout post: rc = 0

00100000:10000000:3.0:1552569516.771127:0:20498:0:(lfsck_layout.c:6395:lfsck_layout_master_in_notify()) testfs-MDT0001-osd: layout LFSCK master handles notify 3 from MDT 0, status 1, flags 0, flags2 0
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;The real root cause is race during set/clear bit operation.&lt;br/&gt;
 lad-&amp;gt;lad_to_post = 0; vs lad-&amp;gt;lad_to_double_scan = 1;&lt;br/&gt;
 And the landed patch avoid unexpected out of order execution didn&apos;t help and probably is wrong. Because set_current_state use barrier already.&lt;br/&gt;
 #define set_current_state(state_value) \&lt;br/&gt;
 set_mb(current-&amp;gt;state, (state_value))&lt;/p&gt;

&lt;p&gt;&#160;I&apos;m reopening ticket and pushing the patch for race.&lt;/p&gt;</comment>
                            <comment id="244660" author="gerrit" created="Tue, 26 Mar 2019 11:46:19 +0000"  >&lt;p&gt;Alexandr Boyko (c17825@cray.com) uploaded a new patch: &lt;a href=&quot;https://review.whamcloud.com/34502&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/34502&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-8760&quot; title=&quot;sanity-lfsck test 31g hung&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-8760&quot;&gt;&lt;del&gt;LU-8760&lt;/del&gt;&lt;/a&gt; lfsck: fix bit operations lfsck_assistant_data&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: 1&lt;br/&gt;
Commit: ce2a56e1aca04d7cf211c50e12e17d34f6587a57&lt;/p&gt;</comment>
                            <comment id="244881" author="panda" created="Fri, 29 Mar 2019 10:40:59 +0000"  >&lt;p&gt;The Linux kernel guarantees that wait_event(..., event_var); and event_var = 1; wake_up(...); are always atomic and consistent and do not require any explicit memory barriers:&lt;/p&gt;
&lt;div class=&quot;code panel&quot; style=&quot;border-width: 1px;&quot;&gt;&lt;div class=&quot;codeContent panelContent&quot;&gt;
&lt;pre class=&quot;code-java&quot;&gt;
SLEEP AND WAKE-UP FUNCTIONS
---------------------------

Sleeping and waking on an event flagged in global data can be viewed as an
interaction between two pieces of data: the task state of the task waiting &lt;span class=&quot;code-keyword&quot;&gt;for&lt;/span&gt;
the event and the global data used to indicate the event.  To make sure that
these appear to happen in the right order, the primitives to begin the process
of going to sleep, and the primitives to initiate a wake up imply certain
barriers.

Firstly, the sleeper normally follows something like &lt;span class=&quot;code-keyword&quot;&gt;this&lt;/span&gt; sequence of events:

        &lt;span class=&quot;code-keyword&quot;&gt;for&lt;/span&gt; (;;) {
                set_current_state(TASK_UNINTERRUPTIBLE);
                &lt;span class=&quot;code-keyword&quot;&gt;if&lt;/span&gt; (event_indicated)
                        &lt;span class=&quot;code-keyword&quot;&gt;break&lt;/span&gt;;
                schedule();
        }

A general memory barrier is interpolated automatically by set_current_state()
after it has altered the task state:

        CPU 1
        ===============================
        set_current_state();
          set_mb();
            STORE current-&amp;gt;state
            &amp;lt;general barrier&amp;gt;
        LOAD event_indicated

set_current_state() may be wrapped by:

        prepare_to_wait();
        prepare_to_wait_exclusive();

which therefore also imply a general memory barrier after setting the state.
The whole sequence above is available in various canned forms, all of which
interpolate the memory barrier in the right place:

        wait_event();
        wait_event_interruptible();
        wait_event_interruptible_exclusive();
        wait_event_interruptible_timeout();
        wait_event_killable();
        wait_event_timeout();
        wait_on_bit();
        wait_on_bit_lock();


Secondly, code that performs a wake up normally follows something like &lt;span class=&quot;code-keyword&quot;&gt;this&lt;/span&gt;:

        event_indicated = 1;
        wake_up(&amp;amp;event_wait_queue);

or:

        event_indicated = 1;
        wake_up_process(event_daemon);

A write memory barrier is implied by wake_up() and co. &lt;span class=&quot;code-keyword&quot;&gt;if&lt;/span&gt; and only &lt;span class=&quot;code-keyword&quot;&gt;if&lt;/span&gt; they wake
something up.  The barrier occurs before the task state is cleared, and so sits
between the STORE to indicate the event and the STORE to set TASK_RUNNING:

        CPU 1                           CPU 2
        =============================== ===============================
        set_current_state();            STORE event_indicated
          set_mb();                     wake_up();
            STORE current-&amp;gt;state          &amp;lt;write barrier&amp;gt;
            &amp;lt;general barrier&amp;gt;             STORE current-&amp;gt;state
        LOAD event_indicated
&lt;/pre&gt;
&lt;/div&gt;&lt;/div&gt;
&lt;p&gt;l_wait_event() is different from wait_event() but seems to mimic the same logic even without the smp_mb() modification.&lt;/p&gt;</comment>
                            <comment id="245353" author="gerrit" created="Mon, 8 Apr 2019 05:32:06 +0000"  >&lt;p&gt;Oleg Drokin (green@whamcloud.com) merged in patch &lt;a href=&quot;https://review.whamcloud.com/34502/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://review.whamcloud.com/34502/&lt;/a&gt;&lt;br/&gt;
Subject: &lt;a href=&quot;https://jira.whamcloud.com/browse/LU-8760&quot; title=&quot;sanity-lfsck test 31g hung&quot; class=&quot;issue-link&quot; data-issue-key=&quot;LU-8760&quot;&gt;&lt;del&gt;LU-8760&lt;/del&gt;&lt;/a&gt; lfsck: fix bit operations lfsck_assistant_data&lt;br/&gt;
Project: fs/lustre-release&lt;br/&gt;
Branch: master&lt;br/&gt;
Current Patch Set: &lt;br/&gt;
Commit: f0ead95dd1275ee906eccdf117abb92b36949a1b&lt;/p&gt;</comment>
                            <comment id="245413" author="mdiep" created="Mon, 8 Apr 2019 14:20:48 +0000"  >&lt;p&gt;Landed in 2.13&lt;/p&gt;</comment>
                            <comment id="344478" author="eaujames" created="Wed, 24 Aug 2022 08:21:24 +0000"  >&lt;p&gt;+1 on b2_12: &lt;a href=&quot;https://testing.whamcloud.com/test_sets/3daca484-09f5-4839-a786-67650dccaf6c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://testing.whamcloud.com/test_sets/3daca484-09f5-4839-a786-67650dccaf6c&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                                                                                                                                            <customfield id="customfield_10890" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10390" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hzytfj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10090" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10060" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Severity</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10022"><![CDATA[3]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        </customfields>
    </item>
</channel>
</rss>