https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8295 Lustre <p>This issue was created by maloo for nasf &lt;fan.yong@intel.com&gt;</p> <p>Please provide additional information about the failure here.</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/dc7b4098-3372-11e6-bbf5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/dc7b4098-3372-11e6-bbf5-5254006e85c2</a>.</p> <p>The logs on the OST show that two mount thread were blocked:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[12811.430006] mount S ffff88007b4e72f0 0 25441 25436 0x00000080 [12811.430006] ffff8800465b7e70 0000000000000086 ffff88007b4e7300 ffff8800465b7fd8 [12811.430006] ffff8800465b7fd8 ffff8800465b7fd8 ffff88007b4e7300 ffff8800465b7ef8 [12811.430006] ffff88007b4e77f0 ffff88007b4e7300 ffff88007b4e7300 ffff88007b4e72f0 [12811.430006] Call Trace: [12811.430006] [&lt;ffffffff8163b349&gt;] schedule+0x29/0x70 [12811.430006] [&lt;ffffffff81080f83&gt;] do_wait+0x203/0x260 [12811.430006] [&lt;ffffffff810820a0&gt;] SyS_wait4+0x80/0x110 [12811.430006] [&lt;ffffffff8107fb20&gt;] ? task_stopped_code+0x60/0x60 [12811.430006] [&lt;ffffffff816463c9&gt;] system_call_fastpath+0x16/0x1b [12811.430006] sh S ffff88007b4e2df0 0 25442 25440 0x00000080 [12811.430006] ffff88007a95fe70 0000000000000086 ffff88007b4e2e00 ffff88007a95ffd8 [12811.430006] ffff88007a95ffd8 ffff88007a95ffd8 ffff88007b4e2e00 ffff88007a95fef8 [12811.430006] ffff88007b4e32f0 ffff88007b4e2e00 ffff88007b4e2e00 ffff88007b4e2df0 [12811.430006] Call Trace: [12811.430006] [&lt;ffffffff8163b349&gt;] schedule+0x29/0x70 [12811.430006] [&lt;ffffffff81080f83&gt;] do_wait+0x203/0x260 [12811.430006] [&lt;ffffffff810820a0&gt;] SyS_wait4+0x80/0x110 [12811.430006] [&lt;ffffffff8107fb20&gt;] ? task_stopped_code+0x60/0x60 [12811.430006] [&lt;ffffffff816463c9&gt;] system_call_fastpath+0x16/0x1b [12811.430006] mount.lustre S 0000000000000000 0 25443 25441 0x00000080 [12811.430006] ffff8800438d7ce0 0000000000000082 ffff8800438c5c00 ffff8800438d7fd8 [12811.430006] ffff8800438d7fd8 ffff8800438d7fd8 ffff8800438c5c00 ffff880047f4c800 [12811.430006] ffff8800438c5c00 ffff8800438d7e08 ffff880043836800 0000000000000000 [12811.430006] Call Trace: [12811.430006] [&lt;ffffffff8163b349&gt;] schedule+0x29/0x70 [12811.430006] [&lt;ffffffffa0abaed6&gt;] server_fill_super+0x1656/0x184c [obdclass] [12811.430006] [&lt;ffffffff810a6ac0&gt;] ? wake_up_atomic_t+0x30/0x30 [12811.430006] [&lt;ffffffffa0a8f9d8&gt;] lustre_fill_super+0x328/0x950 [obdclass] [12811.430006] [&lt;ffffffffa0a8f6b0&gt;] ? lustre_common_put_super+0x270/0x270 [obdclass] [12811.430006] [&lt;ffffffff811e1f2d&gt;] mount_nodev+0x4d/0xb0 [12811.430006] [&lt;ffffffffa0a87908&gt;] lustre_mount+0x38/0x60 [obdclass] [12811.430006] [&lt;ffffffff811e28d9&gt;] mount_fs+0x39/0x1b0 [12811.430006] [&lt;ffffffff811fe1af&gt;] vfs_kern_mount+0x5f/0xf0 [12811.430006] [&lt;ffffffff812006fe&gt;] do_mount+0x24e/0xa40 [12811.430006] [&lt;ffffffff8116e15e&gt;] ? __get_free_pages+0xe/0x50 [12811.430006] [&lt;ffffffff81200f86&gt;] SyS_mount+0x96/0xf0 [12811.430006] [&lt;ffffffff816463c9&gt;] system_call_fastpath+0x16/0x1b [12811.430006] mount S ffff88003dd65070 0 25446 25442 0x00000080 [12811.430006] ffff8800434d3e70 0000000000000086 ffff88003dd65080 ffff8800434d3fd8 [12811.430006] ffff8800434d3fd8 ffff8800434d3fd8 ffff88003dd65080 ffff8800434d3ef8 [12811.430006] ffff88003dd65570 ffff88003dd65080 ffff88003dd65080 ffff88003dd65070 [12811.430006] Call Trace: [12811.430006] [&lt;ffffffff8163b349&gt;] schedule+0x29/0x70 [12811.430006] [&lt;ffffffff81080f83&gt;] do_wait+0x203/0x260 [12811.430006] [&lt;ffffffff810820a0&gt;] SyS_wait4+0x80/0x110 [12811.430006] [&lt;ffffffff8107fb20&gt;] ? task_stopped_code+0x60/0x60 [12811.430006] [&lt;ffffffff816463c9&gt;] system_call_fastpath+0x16/0x1b [12811.430006] mount.lustre S 0000000000000000 0 25449 25446 0x00000080 [12811.430006] ffff88003db0bce0 0000000000000082 ffff88007c141700 ffff88003db0bfd8 [12811.430006] ffff88003db0bfd8 ffff88003db0bfd8 ffff88007c141700 ffff88005146c000 [12811.430006] ffff88007c141700 ffff88003db0be08 ffff88007a89c000 0000000000000000 [12811.430006] Call Trace: [12811.430006] [&lt;ffffffff8163b349&gt;] schedule+0x29/0x70 [12811.430006] [&lt;ffffffffa0abaed6&gt;] server_fill_super+0x1656/0x184c [obdclass] [12811.430006] [&lt;ffffffff810a6ac0&gt;] ? wake_up_atomic_t+0x30/0x30 [12811.430006] [&lt;ffffffffa0a8f9d8&gt;] lustre_fill_super+0x328/0x950 [obdclass] [12811.430006] [&lt;ffffffffa0a8f6b0&gt;] ? lustre_common_put_super+0x270/0x270 [obdclass] [12811.430006] [&lt;ffffffff811e1f2d&gt;] mount_nodev+0x4d/0xb0 [12811.430006] [&lt;ffffffffa0a87908&gt;] lustre_mount+0x38/0x60 [obdclass] [12811.430006] [&lt;ffffffff811e28d9&gt;] mount_fs+0x39/0x1b0 [12811.430006] [&lt;ffffffff811fe1af&gt;] vfs_kern_mount+0x5f/0xf0 [12811.430006] [&lt;ffffffff812006fe&gt;] do_mount+0x24e/0xa40 [12811.430006] [&lt;ffffffff8116e15e&gt;] ? __get_free_pages+0xe/0x50 [12811.430006] [&lt;ffffffff81200f86&gt;] SyS_mount+0x96/0xf0 [12811.430006] [&lt;ffffffff816463c9&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> LU-8295

conf-sanity test_41c: test failed to respond and timed out

Bug Minor Resolved Fixed John Hammond Maloo Fri, 17 Jun 2016 06:45:59 +0000 Sat, 17 Dec 2016 14:50:47 +0000 Sat, 17 Dec 2016 14:50:47 +0000 Lustre 2.10.0 0 5 <p>so OBD_RACE() is supposed to stop for the first caller and then continue when somebody else reaches there.</p> <p>Yet we see both threads stuck in the same place in OBD_RACE? Something strange is going on here. Also there's only one print in the logs about the OBD_RACE actually being hit</p> <p>There's a bug in the test:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #define OBD_FAIL_TGT_MOUNT_RACE 0x716 do_facet ost1 <span class="code-quote">"$LCTL set_param fail_loc=0x716"</span> start ost1 $(ostdevname 1) $OST_MOUNT_OPTS &amp; pid=$! start ost1 $(ostdevname 1) $OST_MOUNT_OPTS &amp; do_facet ost1 <span class="code-quote">"$LCTL set_param fail_loc=0x0"</span> pid2=$! </pre> </div></div> <p>Since both mount commands are asynchronously, we may clear <tt>fail_loc</tt> between the two invocations of <tt>OBD_RACE()</tt>. In this case both threads will hang.</p> <p>&gt; Also there's only one print in the logs about the OBD_RACE actually being hit</p> <p>Shouldn't we expect that CDEBUG rate limitation will prevent the second message from being printed.</p> <p>And we need <tt>CFS_FAIL_ONCE</tt> along with <tt>OBD_FAIL_TGT_MOUNT_RACE</tt> when setting <tt>fail_loc</tt>.</p> <p>John, since I am the "responsible" of this test and associated code, I feel a bit interested by this problem and ticket <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>My understanding was that OBD_RACE()/CFS_RACE()/cfs_race() only/safely work with 2 threads, the 1st stops waiting for the 2nd to awake it.</p> <p>And I think, like you, that for this to work as expected, fail_loc must not be cleared in between, when there is a potential window for this to happen due to the second mount/thread to be run asynchronously, with the clear of fail_loc being launched simultaneously.</p> <p>On the other hand, I thought before that CFS_FAIL_ONCE was useless for OBD_RACE() to work, but after better regarding the code in __cfs_fail_check_set() core routine this may also, as you point, be the reason for both threads being stuck if they are closely racing and have entered the routine at around the same time, and particularly if CFS_FAILED bit has still not been set.</p> <p>Do you want me to push a patch to fix these 2 holes (4 in fact since fail_loc is wrongly set/cleared for MDSs and OSSs cases) in test ?</p> <p>&gt; Do you want me to push a patch to fix these 2 holes (4 in fact since fail_loc is wrongly set/cleared for MDSs and OSSs cases) in test ?</p> <p>Yes, please.</p> <p>I'm still not confident that I understand how this is working but I can see several potential issues with this code: <tt>cfs_race_state</tt> should be declared volatile. There is a race between the two threads setting <tt>cfs_race_state</tt> and one sleeping the other waking. <tt>proc_fail_loc()</tt> calls <tt>wake_up(&amp;cfs_race_waitq)</tt> but it does not set <tt>cfs_race_state</tt> to 1 so it seems like the sleeping task would just go back to sleep.</p> <p>My best guess is that usually "do_facet ost1 $LCTL set_param fail_loc=0x0" is running before either mount command so the <tt>CFS_RACE()</tt> is never triggered.</p> <p>Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21117" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21117</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8295" title="conf-sanity test_41c: test failed to respond and timed out" class="issue-link" data-issue-key="LU-8295"><del>LU-8295</del></a> tests: comply with OBD_RACE() behavior<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e3050d4414667a941c501728de2bfb0b78e8b097</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/21117/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/21117/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8295" title="conf-sanity test_41c: test failed to respond and timed out" class="issue-link" data-issue-key="LU-8295"><del>LU-8295</del></a> tests: comply with OBD_RACE() behavior<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a3becc1092524366e13b3c72d61c6b0ea82709f7</p> <p>Landed for 2.10</p> Development Rank 1|hzyewf: Rank (Obsolete) 9223372036854775807 Severity