https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12100 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;adilger@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/14aef9dc-4ce8-11e9-92fe-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/14aef9dc-4ce8-11e9-92fe-52540065bddc</a></p> <p>test_2 failed with the following errors in the client test output:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>/usr/lib64/lustre/tests/sanity-quota.sh: line 88: zpool: command not found : : User quota (inode hardlimit:1024 files) lfs setquota: warning: inode hardlimit '1024' smaller than minimum qunit size See 'lfs help setquota' or Lustre manual for details : sanity-quota test_2: @@@@@@ FAIL: user create failure, but expect success </pre> </div></div> <p>I haven't investigated closely what the cause is, but this has failed about 10 times in the last week.</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-quota test_2 - user create failure, but expect success</p> LU-12100

sanity-quota test_2: user create failure, but expect success

Bug Minor Resolved Fixed Nathaniel Clark Maloo zfs Sat, 23 Mar 2019 01:49:54 +0000 Sat, 11 Jul 2020 15:22:14 +0000 Wed, 20 May 2020 13:58:54 +0000 Lustre 2.13.0 Lustre 2.12.1 Lustre 2.14.0 Lustre 2.14.0 Lustre 2.12.6 0 13 <p><a href="https://testing.whamcloud.com/test_sets/fd33d7dc-5327-11e9-9720-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/fd33d7dc-5327-11e9-9720-52540065bddc</a></p> <p>This same error made test_33 fail as well...</p> <p>Same error seen in sanity-quota test_3 for 2.12.1 and 2.13.0 with logs at<br/> <a href="https://testing.whamcloud.com/test_sets/81a82ef4-54e6-11e9-b98a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/81a82ef4-54e6-11e9-b98a-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/edda232e-4a56-11e9-a256-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/edda232e-4a56-11e9-a256-52540065bddc</a></p> <p>Another instance failed on test_2:</p> <p><a href="https://testing.whamcloud.com/test_sets/71c71570-71ad-11e9-bd0e-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/71c71570-71ad-11e9-bd0e-52540065bddc</a></p> <p>This appears to be failing only on ZFS. The first similar failure that I could find was on 2018-06-12 with patch <a href="https://review.whamcloud.com/31976" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31976</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-11310" title="support for SLES 15" class="issue-link" data-issue-key="LU-11310"><del>LU-11310</del></a> tests: for test.</tt>", though this is a test patch for SLES15 ldiskfs that never landed, so it is possible there was something wrong with that kernel/patch?<br/> <a href="https://testing.whamcloud.com/test_sets/772bbf04-6dec-11e8-a522-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/772bbf04-6dec-11e8-a522-52540065bddc</a></p> <p>The next failure is on 2018-08-09 with patch <a href="https://review.whamcloud.com/31504" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31504</a> v19 "<tt><a href="https://jira.whamcloud.com/browse/LU-4684" title="DNE3: allow migrating DNE striped directory" class="issue-link" data-issue-key="LU-4684"><del>LU-4684</del></a> lmv: support accessing migrating directory</tt>", which eventually landed on 2018-10-01, which is after the <em>next</em> failure on 2018-08-23, so that could not have been the patch that introduced the problem.</p> <p>I'd be inclined to look for a patch that landed just before 2018-08-09 (there was a big batch of landings on 2018-08-06), which has patch <a href="https://review.whamcloud.com/32827" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32827</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-11153" title="the uninitialized variable &quot;ver&quot; could be used in qmt_set" class="issue-link" data-issue-key="LU-11153"><del>LU-11153</del></a> quota: initialize ver for default quota</tt>" among others, and more on 2018-08-09.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=stancheff" class="user-hover" rel="stancheff">stancheff</a>, you comment was correct, and looks to be useful!  I'm going to push a patch. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35577" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35577</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12100" title="sanity-quota test_2: user create failure, but expect success " class="issue-link" data-issue-key="LU-12100"><del>LU-12100</del></a> tests: Fix typo in project quota support<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 647cb8165588d8171a0cb1d3095bc673a53f8efd</p> <p>Nathaniel Clark (nclark@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35667" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35667</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12100" title="sanity-quota test_2: user create failure, but expect success " class="issue-link" data-issue-key="LU-12100"><del>LU-12100</del></a> tests: Use minimum soft qunit limit<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 06f63234459f92d8ff5186252ac61457fb6b7e02</p> <p>ZFS 0.8.0 failures:<br/> <a href="https://testing.whamcloud.com/test_sets/b4d07d78-b4d5-11e9-9f36-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b4d07d78-b4d5-11e9-9f36-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/09cf7628-b28b-11e9-bcf0-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/09cf7628-b28b-11e9-bcf0-52540065bddc</a></p> <p>The issue is that the second dd ("Write to exceed soft limit") seems to run much slower than first dd, and exceeds the grace period.</p> <p>First dd is between 65 and 200MB/s</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Write up to soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d3.sanity-quota/f3.sanity-quota-0] [count=4] 4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied, 0.0603464 s, 69.5 MB/s </pre> </div></div> <p>The second is &gt; 0.5MB/s</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Write to exceed soft limit running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d3.sanity-quota/f3.sanity-quota-0] [bs=1K] [count=10] [seek=4096] 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 33.0001 s, 0.3 kB/s </pre> </div></div> <p>When the third ("Write before timer goes off") attempts to write the grace timer has in fact already expired.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 5125* 4096 0 none 1 0 0 - </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Write before timer goes off running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d3.sanity-quota/f3.sanity-quota-0] [bs=1K] [count=10] [seek=5120] dd: error writing '/mnt/lustre/d3.sanity-quota/f3.sanity-quota-0': Disk quota exceeded </pre> </div></div> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35667/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35667/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12100" title="sanity-quota test_2: user create failure, but expect success " class="issue-link" data-issue-key="LU-12100"><del>LU-12100</del></a> tests: Use minimum soft qunit limits<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 37e28b7e05a5b1f77fe663f9407436aea312b3b2</p> <p>Landed for 2.13</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36346" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36346</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12100" title="sanity-quota test_2: user create failure, but expect success " class="issue-link" data-issue-key="LU-12100"><del>LU-12100</del></a> tests: Use minimum soft qunit limits<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: a42726bd0bc203163644de63705dc3d148a54a0e</p> <p>Still seeing this on latest master (2.13.50):<br/> <a href="https://testing.whamcloud.com/test_sets/8fa3c85c-07e1-11ea-8e77-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8fa3c85c-07e1-11ea-8e77-52540065bddc</a></p> <p>Nathaniel Clark (nclark@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36797" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36797</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12100" title="sanity-quota test_2: user create failure, but expect success " class="issue-link" data-issue-key="LU-12100"><del>LU-12100</del></a> tests: Use least qunit to set limit<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 38306a6354314cc0b59ae180f311069f8a67796e</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36346/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36346/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12100" title="sanity-quota test_2: user create failure, but expect success " class="issue-link" data-issue-key="LU-12100"><del>LU-12100</del></a> tests: Use minimum soft qunit limits<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 17802fc3336b0a9546a03dd10efa496f6a7f6f42</p> <p><a href="https://testing.whamcloud.com/test_sets/efbb05c6-3b34-11ea-971c-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/efbb05c6-3b34-11ea-971c-52540065bddc</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/df3f8624-4f73-11ea-a90e-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/df3f8624-4f73-11ea-a90e-52540065bddc</a></p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36797/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36797/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12100" title="sanity-quota test_2: user create failure, but expect success " class="issue-link" data-issue-key="LU-12100"><del>LU-12100</del></a> tests: Use least qunit to set limit<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 33e500cfb33406b8dddac46e1dfb5a3d59ff01c5</p> <p>Landed for 2.14</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38769" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38769</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12100" title="sanity-quota test_2: user create failure, but expect success " class="issue-link" data-issue-key="LU-12100"><del>LU-12100</del></a> tests: Use least qunit to set limit<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: c6985624f377888ba134883b742c07e36bd3bae2</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38769/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38769/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12100" title="sanity-quota test_2: user create failure, but expect success " class="issue-link" data-issue-key="LU-12100"><del>LU-12100</del></a> tests: Use least qunit to set limit<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: abb6af1ab8f74df8d0aa1e728c63fe67b7b2d3e1</p> Blocker LU-12336 Related LU-11544 LU-13639 Development Rank 1|i00ds7: Rank (Obsolete) 9223372036854775807 Severity