https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10570 Lustre <p>== sanity test 27y: create files while OST0 is degraded and the rest inactive ======================== 06:05:43 (1516946743)<br/> CMD: onyx-37vm9 lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_last_id<br/> CMD: onyx-37vm9 lctl get_param -n osc.lustre-OST0000-osc-MDT0000.prealloc_next_id<br/> CMD: onyx-37vm9 lctl dl<br/> lustre-OST0001-osc-MDT0000 is Deactivated:<br/> CMD: onyx-37vm9 lctl --device %lustre-OST0001-osc-MDT0000 deactivate<br/> lustre-OST0000 is degraded:<br/> CMD: onyx-37vm8 lctl set_param -n obdfilter.lustre-OST0000.degraded=1<br/> CMD: onyx-37vm9 lctl get_param -n lov.*.qos_maxage<br/> total: 2 open/close in 0.00 seconds: 437.82 ops/second<br/> lustre-OST0000 is recovered from degraded:<br/> CMD: onyx-37vm8 lctl set_param -n obdfilter.lustre-OST0000.degraded=0<br/> CMD: onyx-37vm9 lctl --device %lustre-OST0001-osc-MDT0000 activate<br/> CMD: onyx-37vm9 lctl get_param -n lov.*.qos_maxage<br/> sanity test_27y: @@@@@@ FAIL: Of 2 OSTs, only 1 is available <br/> Trace dump:<br/> = /usr/lib64/lustre/tests/test-framework.sh:5336:error()<br/> = /usr/lib64/lustre/tests/sanity.sh:1840:test_27y()<br/> = /usr/lib64/lustre/tests/test-framework.sh:5612:run_one()<br/> = /usr/lib64/lustre/tests/test-framework.sh:5651:run_one_logged()<br/> = /usr/lib64/lustre/tests/test-framework.sh:5498:run_test()<br/> = /usr/lib64/lustre/tests/sanity.sh:1843:main()</p> <p>This issue was created by maloo for Jinshan Xiong &lt;jinshan.xiong@intel.com&gt;</p> <p>This issue relates to the following test suite run: </p> <p>&lt;&lt;Please provide additional information about the failure here&gt;&gt;</p> LU-10570

sanity test_27y: Error: 'Of 2 OSTs, only 1 is available'

Bug Minor Resolved Fixed James A Simmons Maloo Fri, 26 Jan 2018 18:34:37 +0000 Tue, 27 Feb 2018 04:35:17 +0000 Tue, 27 Feb 2018 04:35:17 +0000 Lustre 2.11.0 Lustre 2.11.0 0 11 <p>This issue should be introduced by a recent commit because I have never seen it before</p> <p>We've seen this test fail four times since January 26, 2017 and all of those failures are in review-zfs. Here are links to some of the logs for these failures:<br/> <a href="https://testing.hpdd.intel.com/test_sets/7e9510c2-050c-11e8-a10a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7e9510c2-050c-11e8-a10a-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/4ba8b3e8-0486-11e8-a7cd-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4ba8b3e8-0486-11e8-a7cd-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/de800ac0-02da-11e8-a6ad-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/de800ac0-02da-11e8-a6ad-52540065bddc</a></p> <p>Bisection shows that this was introduced by <a href="https://review.whamcloud.com/30867" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30867</a> <a href="https://jira.whamcloud.com/browse/LU-9019" title="Migrate lustre to standard 64 bit time kernel API" class="issue-link" data-issue-key="LU-9019"><del>LU-9019</del></a> libcfs: remove cfs_time_XXX_64 wrappers.</p> <p>James</p> <p>Should we revert this change or is it possible to easily fix this issue?</p> <p>Peter</p> <p>It should easy to fix. I have a feeling its one of those using seconds resolution is not good enough. I bet only seen on VMs.</p> <p>Can't reproduce on ldiskfs locally. Attempted to bring up ZFS for test suite but I have run into some issues with setup. I get device is to small. smaller than 64MB issues. Looking I don't see any docs on how to setup the test suite with ZFS. Any pointers? Can anyone reproduce this easily?</p> <p>Saw the same issue before. From my side, it worked after I just used dd to write a bigger Lustre target files of <tt>/tmp/lustre-{ost,mdt}X</tt>, like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">dd <span class="code-keyword">if</span>=/dev/zero of=/tmp/lustre-mdt1 bs=1M count=1 seek=8191 </pre> </div></div> <p>James,</p> <p>I was able to reproduce this using a llmount.sh filesystem with FSTYPE=ldiskfs on a single RHEL 7.4 VM. It didn't fail every time but it would consistently fail within a minute when I ran <tt>while ONLY=27y bash lustre/tests/sanity.sh; do true; done</tt>.</p> <p>For the FSTYPE=zfs issue, see <a href="https://jira.whamcloud.com/browse/LU-10424" title="FSTYPE=zfs bash lustre/tests/llmount.sh fails unless /tmp/lustre-{mdt1,ost1,ost2} all exist and have size at least 64M" class="issue-link" data-issue-key="LU-10424">LU-10424</a>.</p> <p>I have a patch cooked up. Will push after I'm done testing.</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/31127" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31127</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10570" title="sanity test_27y: Error: &#39;Of 2 OSTs, only 1 is available&#39;" class="issue-link" data-issue-key="LU-10570"><del>LU-10570</del></a> obd: use ktime_t for statfs handling<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e4441d88f18882bf3b1441ddf02ea59f843ca207</p> <p>James - can you explain why ktime_t change would cause the problem?</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/31158" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31158</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10570" title="sanity test_27y: Error: &#39;Of 2 OSTs, only 1 is available&#39;" class="issue-link" data-issue-key="LU-10570"><del>LU-10570</del></a> obd: fix statfs handling<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b6cd8307c488cffe2bbd5819b42583ab340610f7</p> <p>So in the original code handling the stat refreshing was using 64 bit jiffies which means we usually have time resolution in the milliseconds. Because it was the 64 bit version of jiffies didn't mean it has better time resolution like ktime_t does. Looking at the code it feels natural to use seconds resolution since OBD_STATFS_CACHE_SECONDS is one second and the qos max_age is also in seconds which is why I moved in that direction. Also the comments above obd_statfs() pointed to a more second resolution approach.  What is causing the pain is that in lod_qos_statfs_updates() we test twice if the stats need to be refreshed due to the millisecond resolution. One before lq_rw_sem is taken and then again after taking the semaphore. For the case of using jiffies level resolution the chances that condition one is false and condition two is true is pretty slim. When the code moved to using time64_t that change greatly increased.</p> <p>So I have approach this problem in two ways. First one was to move to ktime_t and maintain the original behavior of the code. The second was to remove the second test for the need to refresh the cache which seems to work. I have both options posted seen their might be other behavior  changes with the removal of the second stale stats test in lod_qos_statfs_updates(). We can ponder which is the better approach.</p> <p>+1 on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/28ef2abe-0ccc-11e8-a7cd-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/28ef2abe-0ccc-11e8-a7cd-52540065bddc</a></p> <p>I have a fix at <a href="https://review.whamcloud.com/#/c/31158/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/31158/</a></p> <p>more on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/497dd194-0d2c-11e8-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/497dd194-0d2c-11e8-bd00-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/2c909e06-0dbb-11e8-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2c909e06-0dbb-11e8-bd00-52540065bddc</a></p> <p>more on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/9ff21932-17f3-11e8-a7cd-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9ff21932-17f3-11e8-a7cd-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/9ea2f9ec-1808-11e8-a10a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9ea2f9ec-1808-11e8-a10a-52540065bddc</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/7bcf5f50-1817-11e8-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7bcf5f50-1817-11e8-bd00-52540065bddc</a></p> <p>Patch is in master-next so the fix should land soon.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31158/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31158/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10570" title="sanity test_27y: Error: &#39;Of 2 OSTs, only 1 is available&#39;" class="issue-link" data-issue-key="LU-10570"><del>LU-10570</del></a> obd: fix statfs handling<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 87577f4988c1814dae1a1274880e20f1991e7b94</p> <p>Landed for 2.11</p> Related LU-9019 LU-10576 Development Rank 1|hzzrsn: Rank (Obsolete) 9223372036854775807 Severity