https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5638 Lustre <p>This issue was created by maloo for nasf &lt;fan.yong@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/a26efad0-3e95-11e4-916a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a26efad0-3e95-11e4-916a-5254006e85c2</a>.</p> <p>The sub-test test_33 failed with the following error:</p> <blockquote> <p>Used inodes for user 60000 isn't 0. 1</p></blockquote> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: sanity-quota 33</p> LU-5638

sanity-quota test_33 for ZFS-based backend: Used inodes for user 60000 isn't 0. 1

Bug Major Closed Fixed Hongchao Zhang Maloo zfs Thu, 18 Sep 2014 01:47:11 +0000 Thu, 2 Aug 2018 20:17:28 +0000 Thu, 2 Aug 2018 20:17:28 +0000 Lustre 2.7.0 Lustre 2.12.0 0 17 <p>Lots of errors like these in the debug logs:<br/> 00000001:04000000:0.0:1410962344.043221:0:30818:0:(osd_quota.c:98:osd_acct_index_lookup()) lustre-OST0001: id ea60 not found in DMU accounting ZAP<br/> 00000001:04000000:0.0:1410962344.043236:0:30818:0:(osd_quota.c:117:osd_acct_index_lookup()) lustre-OST0001: id ea60 not found in accounting ZAP</p> <p>And 60000=0xea60, so it looked like the user hadn't created anything yet so the zap_lookup() returned negative ENOENT for both the DMU ZAP and the OSD ZAP. But in this case osd_acct_index_lookup() already sets both rec-&gt;bspace and rec-&gt;ispace to 0. I'm a bit confused by the returns of osd_acct_index_lookup() though: it returns either +1 or -errno, but lquota_disk_read() callers expect 0 for success -ENOENT and others for error.</p> <p>Someone who knows the quota code should comment.</p> <blockquote> <p> I'm a bit confused by the returns of osd_acct_index_lookup() though: it returns either +1 or -errno, but lquota_disk_read() callers expect 0 for success -ENOENT and others for error.</p></blockquote> <p>dt_lookup() converted the return values.</p> <blockquote> <p>Lots of errors like these in the debug logs:<br/> 00000001:04000000:0.0:1410962344.043221:0:30818:0:(osd_quota.c:98:osd_acct_index_lookup()) lustre-OST0001: id ea60 not found in DMU accounting ZAP<br/> 00000001:04000000:0.0:1410962344.043236:0:30818:0:(osd_quota.c:117:osd_acct_index_lookup()) lustre-OST0001: id ea60 not found in accounting ZAP</p> <p>And 60000=0xea60, so it looked like the user hadn't created anything yet so the zap_lookup() returned negative ENOENT for both the DMU ZAP and the OSD ZAP. But in this case osd_acct_index_lookup() already sets both rec-&gt;bspace and rec-&gt;ispace to 0. </p></blockquote> <p>These messages were from OSTs, and the test was failed for incorrect inode usage (which is inode usage on MDT), so I think those OST messages is irrelevant.</p> <p>I checked MDT log, but didn't find anything abnormal. I suspect this failure is caused by the race of updating inode accounting ZAP: zap_increment_int() doesn't take lock to make "lookup -&gt; update" atomic. I believe the patch from <a href="https://jira.whamcloud.com/browse/LU-2600" title="lustre metadata performance is very slow on zfs" class="issue-link" data-issue-key="LU-2600"><del>LU-2600</del></a> could fix this problem, unfortunately, the patch was reverted for some regression. I'm not sure if Alex is still working this.</p> <p>As a short term solution, probably we may introduce some lock in osd layer to serialize the zap_increment_int()?</p> <p>I think it makes sense to fix zap_increment_int() instead - it needs exclusive access to do zap_update() anyway.</p> <p>doing so on every accounting change would be very expensive, IMO. instead we should be doing this at commit where all "users" transactions are done, when we have an exclusive access by definition.</p> <p>Yes of course, batching the updates at sync time would be the best solution. Actually that's exactly how DMU updates the DMU_USERUSED_OBJECT/DMU_GROUPUSED_OBJECT, in dsl_pool_sync()<del>&gt;dmu_objset_do_userquota_updates()</del>&gt;do_userquota_update()-&gt;zap_increment_int().</p> <p>right, this is what I was trying to implement in <a href="http://review.whamcloud.com/#/c/10785/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10785/</a>, but failed.</p> <p>Johann has asked me to work on adding dnode accounting support to ZFS in <a href="https://jira.whamcloud.com/browse/LU-2435" title="inode accounting in osd-zfs is racy" class="issue-link" data-issue-key="LU-2435"><del>LU-2435</del></a>.</p> <p>Another instance of this failure at <a href="https://testing.hpdd.intel.com/test_sets/2caf1f82-1f45-11e5-a4d6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2caf1f82-1f45-11e5-a4d6-5254006e85c2</a></p> <p>These tests seem hitting occurences of same issue (sanity quota 33, 34 and 35):</p> <ul> <li><a href="https://testing.hpdd.intel.com/test_logs/63ba8d80-21ad-11e5-a979-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_logs/63ba8d80-21ad-11e5-a979-5254006e85c2</a></li> <li><a href="https://testing.hpdd.intel.com/test_logs/630f5262-21ad-11e5-a979-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_logs/630f5262-21ad-11e5-a979-5254006e85c2</a></li> <li><a href="https://testing.hpdd.intel.com/test_logs/63384fdc-21ad-11e5-a979-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_logs/63384fdc-21ad-11e5-a979-5254006e85c2</a></li> </ul> <p>two new occurrences on master<br/> <a href="https://testing.hpdd.intel.com/test_sessions/761c8276-2419-11e5-91e9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/761c8276-2419-11e5-91e9-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/1b40755e-242a-11e5-87f6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/1b40755e-242a-11e5-87f6-5254006e85c2</a></p> <p>sanity-quota test 11 started failing less than a week ago with inode quota issues. The test is failing with </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Used inodes(1) is less than 2 </pre> </div></div> <p>It looks like the test 11 failures might be the same or related to this ticket because the MDS debug_log contains the same messages as above:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(osd_quota.c:120:osd_acct_index_lookup()) lustre-MDT0000: id ea60 not found in DMU accounting ZAP </pre> </div></div> <p>In the cases below, sanity-quota tests 33, 34 and 35 all fail after test 11 fails:<br/> 2015-07-03 12:14:19 - <a href="https://testing.hpdd.intel.com/test_sets/5ee4d41e-21ad-11e5-a979-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5ee4d41e-21ad-11e5-a979-5254006e85c2</a><br/> 2015-07-03 19:34:12 - <a href="https://testing.hpdd.intel.com/test_sets/41c9e6e6-21e7-11e5-b398-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/41c9e6e6-21e7-11e5-b398-5254006e85c2</a><br/> 2015-07-05 17:43:49 - <a href="https://testing.hpdd.intel.com/test_sets/ff5d7856-2365-11e5-a6ad-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ff5d7856-2365-11e5-a6ad-5254006e85c2</a><br/> 2015-07-06 17:15:26 - <a href="https://testing.hpdd.intel.com/test_sets/7a443f68-242a-11e5-87f6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7a443f68-242a-11e5-87f6-5254006e85c2</a><br/> 2015-07-06 21:12:05 - <a href="https://testing.hpdd.intel.com/test_sets/080b7f58-244b-11e5-91e9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/080b7f58-244b-11e5-91e9-5254006e85c2</a></p> <p>3 new+consecutive occurences for the same master patch (<a href="http://review.whamcloud.com/14384/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14384/</a>) review :<br/> <a href="https://testing.hpdd.intel.com/test_sessions/da77e524-21e6-11e5-b398-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/da77e524-21e6-11e5-b398-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/0a6988f4-2585-11e5-a6b1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/0a6988f4-2585-11e5-a6b1-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/72ee971e-2627-11e5-8b33-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/72ee971e-2627-11e5-8b33-5254006e85c2</a><br/> is there any activity ongoing for this issue?</p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15590" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15590</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5638" title="sanity-quota test_33 for ZFS-based backend: Used inodes for user 60000 isn&#39;t 0. 1" class="issue-link" data-issue-key="LU-5638"><del>LU-5638</del></a> tests: Skip sanity-quota 11 and 33 for ZFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ff39144cf538fc5cac88b1ad61d16640b7854b09</p> <p>Temporarily skipping sanity-quota tests 11 and 33 for review-zfs-part-* until the patch for <a href="https://jira.whamcloud.com/browse/LU-2435" title="inode accounting in osd-zfs is racy" class="issue-link" data-issue-key="LU-2435"><del>LU-2435</del></a> lands.</p> <p>Patch at <a href="http://review.whamcloud.com/#/c/15590" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15590</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/e8dfb4f0-2c15-11e5-8c67-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e8dfb4f0-2c15-11e5-8c67-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15590/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15590/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5638" title="sanity-quota test_33 for ZFS-based backend: Used inodes for user 60000 isn&#39;t 0. 1" class="issue-link" data-issue-key="LU-5638"><del>LU-5638</del></a> tests: Skip sanity-quota tests for ZFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 87a28e4280e08947d852a317f49be174cc6f4cb6</p> <p>Landed for 2.8</p> <p>This issue is not resolved. Only a patch to skip the tests was landed. The original problem causing sanity-quota 11, 33, 34, and 35 still exists.</p> <p>There is a belief that this was caused by slow ZFS metadata performance, which has been improved in Lustre 2.9. It would be worthwhile to retest these skipped tests (with ZFS of course) to see if they now pass reliably.</p> <p>being seen in non-zfs tests too. example:<br/> <a href="https://testing.hpdd.intel.com/test_sets/834ddd66-472e-11e7-b3fe-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/834ddd66-472e-11e7-b3fe-5254006e85c2</a></p> <p>I note that test 33 is skipped with ALWAYS_EXCEPT for test runs on zfs. Maybe it needs to be skipped all the time on everything.</p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/1637dff4-4839-11e7-bc6c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1637dff4-4839-11e7-bc6c-5254006e85c2</a></p> <p>I don't think skipping the test is the right way forward, except as a short-term workaround. Instead, someone needs to take the time to figure out what file is being left behind with this UID. </p> <p>I think the old issue should have been fixed once the <a href="https://jira.whamcloud.com/browse/LU-2435" title="inode accounting in osd-zfs is racy" class="issue-link" data-issue-key="LU-2435"><del>LU-2435</del></a> being landed, we can re-enable test_33 for zfs testing now, I'll cook a patch to re-enable it.</p> <p>The new occurrences on ldiskfs is another issue, I believe it's a defect in project quota:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">sanity-quota test_33: @@@@@@ FAIL: Used space <span class="code-keyword">for</span> project 1000:18432, expected:20480 </pre> </div></div> <p>I think we'd open a new ticket for it.</p> <p>The new issue is created at <a href="https://jira.whamcloud.com/browse/LU-9597" title="sanity-quota test_33: &#39;Used space for project 1000:18432, expected:20480&#39;" class="issue-link" data-issue-key="LU-9597"><del>LU-9597</del></a>.</p> <p>Niu Yawei (yawei.niu@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/27423" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27423</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5638" title="sanity-quota test_33 for ZFS-based backend: Used inodes for user 60000 isn&#39;t 0. 1" class="issue-link" data-issue-key="LU-5638"><del>LU-5638</del></a> tests: re-enable zfs quota tests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ec72f9a800214354e537b3f95c83c1ea509fa178</p> <p>It looks like sanity-quota test 33 is still failing with ZFS servers. </p> <p>Logs for two recent failures are at:<br/> <a href="https://testing.hpdd.intel.com/test_sets/506f3d2e-480d-11e7-91f4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/506f3d2e-480d-11e7-91f4-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/ac775cfe-4a84-11e7-91f4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ac775cfe-4a84-11e7-91f4-5254006e85c2</a></p> <p>sanity-quota test 33 is failing. Maloo link to look at needed information</p> <p><a href="https://testing.hpdd.intel.com/test_sets/8442a52c-7bad-11e7-a168-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8442a52c-7bad-11e7-a168-5254006e85c2</a></p> <p>Error: 'Used inode for user 60000 is 1, expected 10' <br/> Wait for setattr on objects finished...<br/> sleep 5 for ZFS OSD<br/> Waiting for local destroys to complete<br/> CMD: onyx-45vm7,onyx-45vm8 lctl set_param -n osd*.<b>MDT</b>.force_sync=1<br/> CMD: onyx-45vm10 lctl set_param -n osd*.<b>OS</b>.force_sync=1<br/> Verify disk usage after write<br/> Verify inode usage after write<br/> sanity-quota test_33: @@@@@@ FAIL: Used inode for user 60000 is 1, expected 10 <br/> Trace dump:<br/> = /usr/lib64/lustre/tests/test-framework.sh:5291:error()<br/> = /usr/lib64/lustre/tests/sanity-quota.sh:2423:test_33()<br/> = /usr/lib64/lustre/tests/test-framework.sh:5567:run_one()<br/> = /usr/lib64/lustre/tests/test-framework.sh:5606:run_one_logged()<br/> = /usr/lib64/lustre/tests/test-framework.sh:5453:run_test()<br/> = /usr/lib64/lustre/tests/sanity-quota.sh:2450:main()<br/> Dumping lctl log to /test_logs/2017-08-07/lustre-reviews-el7-x86_64-<del>custom</del>-1_101_1_<em>49353</em>_-70097615897520-184409/sanity-quota.test_33.*.1502136383.log<br/> CMD: onyx-45vm10,onyx-45vm1.onyx.hpdd.intel.com,onyx-45vm2,onyx-45vm7,onyx-45vm8 /usr/sbin/lctl dk &gt; /test_logs/2017-08-07/lustre-reviews-el7-x86_64-<del>custom</del>-1_101_1_<em>49353</em>_-70097615897520-184409/sanity-quota.test_33.debug_log.\$(hostname -s).1502136383.log;<br/> dmesg &gt; /test_logs/2017-08-07/lustre-reviews-el7-x86_64-<del>custom</del>-1_101_1_<em>49353</em>_-70097615897520-184409/sanity-quota.test_33.dmesg.\$(hostname -s).1502136383.log<br/> Resetting fail_loc on all nodes...CMD: onyx-45vm10,onyx-45vm1.onyx.hpdd.intel.com,onyx-45vm2,onyx-45vm7,onyx-45vm8 lctl set_param -n fail_loc=0 fail_val=0 2&gt;/dev/null<br/> done.</p> <p>Hongchao</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>there is no abnormal information in the logs, and it could still be related to the ZFS performance.<br/> this issue has not occurred since Dec 19, 2017</p> <p>It appears that this was "fixed" by the landing of <a href="https://review.whamcloud.com/27093" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27093</a> which changed the detection of ZFS project quotas but broke detection of ZFS dnode accounting. That patch landed to b2_10 on Dec 20, 2017 (master landing on Nov 9, 2017).</p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32694" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32694</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5638" title="sanity-quota test_33 for ZFS-based backend: Used inodes for user 60000 isn&#39;t 0. 1" class="issue-link" data-issue-key="LU-5638"><del>LU-5638</del></a> tests: resume running sanity-quota tests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 78fee01723aece184b4e328335bab5e120667583</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32694/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32694/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5638" title="sanity-quota test_33 for ZFS-based backend: Used inodes for user 60000 isn&#39;t 0. 1" class="issue-link" data-issue-key="LU-5638"><del>LU-5638</del></a> tests: resume running sanity-quota tests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2fa651d7c9bbeb2ce87e149f25d753c7b66640ab</p> <p>Patch landed to remove sanity-quota 33 from the ALWAYS_EXCEPT list for 2.11.54.</p> Related LU-11024 LU-2435 LU-7991 LU-9592 Development Rank 1|hzwwlr: Rank (Obsolete) 15788 Severity