https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8999 Lustre <p>This issue was created by maloo for Saurabh Tandan &lt;saurabh.tandan@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/d6ff8e38-d464-11e6-ace4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d6ff8e38-d464-11e6-ace4-5254006e85c2</a>.</p> <p>The sub-test test_38 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>skipped id entries </pre> </div></div> <p>test_logs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity-quota test 38: Quota accounting iterator doesn't skip id entries =========================== 05:03:51 (1483707831) Waiting for local destroys to complete Creating test directory CMD: onyx-35vm3,onyx-35vm4 lctl set_param fail_val=0 fail_loc=0 fail_val=0 fail_loc=0 fail_val=0 fail_loc=0 Create 10000 files... CMD: onyx-35vm3 lctl set_param -n osd*.*MDT*.force_sync=1 CMD: onyx-35vm4 lctl set_param -n osd*.*OS*.force_sync=1 CMD: onyx-35vm3 /usr/sbin/lctl get_param osd-ldiskfs.lustre-MDT0000.quota_slave.acct_user Found 10010 id entries sanity-quota test_38: @@@@@@ FAIL: skipped id entries </pre> </div></div> RHEL 7.3 Server/Client<br/> master, build# 3486 LU-8999

sanity-quota test_38: skipped id entries

Bug Major Resolved Fixed Hongchao Zhang Maloo Tue, 10 Jan 2017 21:47:00 +0000 Mon, 21 Jan 2019 15:21:57 +0000 Wed, 8 Aug 2018 20:42:30 +0000 Lustre 2.10.0 Lustre 2.10.1 Lustre 2.11.0 Lustre 2.10.2 Lustre 2.12.0 Lustre 2.10.7 0 10 <p>Another failure on master:</p> <p><a href="https://testing.hpdd.intel.com/test_sets/0fa1f43e-e8c8-11e6-935d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0fa1f43e-e8c8-11e6-935d-5254006e85c2</a></p> <p>2.9.57, b3575:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/2209839b-42d4-4fe6-91f1-96f9ce3c5a69" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/2209839b-42d4-4fe6-91f1-96f9ce3c5a69</a></p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28345" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28345</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> quota: fix quota iteration interface<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fa22284aed4ffdfc9d56360a2aa11f8bcb70b109</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28530" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28530</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> quota: fix quota iteration interface<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 7b65a74570a9c9c047c4165667a55c7ae9c85a54</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28345/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28345/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> quota: fix quota iteration interface<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e431666d6636ce5ff3637592188fc96bcf695a31</p> <p>Landed for 2.11</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/28530/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28530/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> quota: fix quota iteration interface<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: c9df32207c47f71f9840ed2e68cd23612b205ffb</p> <p>still hit this problem on b2_10 2.10.1 RC1 testing RHEL7.4 zfs<br/> <a href="https://testing.hpdd.intel.com/test_sets/fff7634e-9c9e-11e7-ba27-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fff7634e-9c9e-11e7-ba27-5254006e85c2</a></p> <p>another one on 2.10.1 after the patch landed<br/> <a href="https://testing.hpdd.intel.com/test_sets/c81ce8c8-9c66-11e7-ba27-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c81ce8c8-9c66-11e7-ba27-5254006e85c2</a></p> <p>The original failure occurred in ldiskfs test session, while the patch fixed ZFS related codes.<br/> The failure still exists on master branch, here is one more failure instance in ldiskfs test session:<br/> <a href="https://testing.hpdd.intel.com/test_sets/3eff0f0a-9ef8-11e7-b778-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3eff0f0a-9ef8-11e7-b778-5254006e85c2</a></p> <p>Hongchao -</p> <p>Would you please comment on this issue?</p> <p>Thank you.</p> <p>status update:<br/> I have investigated the related logs and code lines, but no progress yet. will update its status once there is any progress.</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30145" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30145</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> test: debug patch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b122751e61f678b807fa87e686b90b27969cd8b2</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30243" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30243</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> test: check chown result<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e6da5d9b7c257dd24314feae85f86692ce8d9203</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30243/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30243/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> test: check chown result<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 817b367707d0d4f5c406b4f18cffceea2aae8406</p> <p>It looks to me like a correction for the test has landed and I expect the debug patch will now be abandoned.</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30631" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30631</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> test: check chown result<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 34983c6f01c974f10ba5b213a60b4a6ddf8ced88</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/30631/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30631/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> test: check chown result<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 3a0fc1a8d65d4796a3da895006dcc5012e9e1b3f</p> <p>doesn't look like this is fixed<br/> <a href="https://testing.hpdd.intel.com/test_sets/8ac15c2e-f1a3-11e7-8c23-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8ac15c2e-f1a3-11e7-8c23-52540065bddc</a></p> <p>I have checked the log, there are 13 extra quota id, which could be left by other tests, but the quota id 0 ~ 9999 required by<br/> this test was there and no problem.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@zhanghc ~]#grep "id:" quota_ids |awk '{print $3}'|sort -n 0 1 ... 9998 9999 10273 14421 18394 19745 20974 23221 23332 25594 27020 29911 30457 30979 31362 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>- id: 10273 usage: { inodes: 1, kbytes: 4 } - id: 14421 usage: { inodes: 1, kbytes: 4 } - id: 18394 usage: { inodes: 1, kbytes: 0 } - id: 19745 usage: { inodes: 1, kbytes: 4 } - id: 20974 usage: { inodes: 1, kbytes: 4 } - id: 23221 usage: { inodes: 1, kbytes: 0 } - id: 23332 usage: { inodes: 1, kbytes: 0 } - id: 25594 usage: { inodes: 1, kbytes: 4 } - id: 27020 usage: { inodes: 1, kbytes: 0 } - id: 29911 usage: { inodes: 1, kbytes: 0 } - id: 30457 usage: { inodes: 1, kbytes: 0 } - id: 30979 usage: { inodes: 1, kbytes: 4 } - id: 31362 usage: { inodes: 1, kbytes: 0 } </pre> </div></div> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30730" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30730</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> test: ignore unrelated quota id<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e0d615954ba8eb1303e20296c164c30fca5e3ebe</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30730/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30730/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> test: ignore unrelated quota id<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8a0a9190cd0a68a88e42210e9c3dac324319fa6d</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30896" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30896</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> test: ignore unrelated quota id<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 01a8b2e0d8afa058e62cac1d24c8b520b81f9671</p> <p>Reopening the issue as it still occurs for 2.10.57 "SLES 12 SP3 Server/DNE/ldiskfs SLES 12 SP3 Client" config.<br/> <a href="https://testing.hpdd.intel.com/test_sets/acaba252-fd4e-11e7-a7cd-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/acaba252-fd4e-11e7-a7cd-52540065bddc</a><br/> Looks like it didn't found enough id entries</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31098" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31098</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> test: debug patch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 30f1ac0972d068350cb1cd10146f596bf14814df</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/30896/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30896/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> test: ignore unrelated quota id<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: b5073eb0234e02892d22c5dead4a3a8a3da065bd</p> <p>I have managed to reproduce this issue in my local VMs, and there are two problems to cause it.</p> <p>1. the problem caused by multiple calls of the "lprocfs_quota_seq_start" of the seq_operations<br/> During the reading of the sequence file, "seq_read" will be called multiple times to read the content,<br/> and start/stop operation will be called in each call of "seq_read", but the start operation will cause<br/> the following problem,<br/> Assume the last stop operation save the current read position at 2000 (qid), the saved block and index is<br/> <span class="error">&#91;1, 2, 3, 100, 200&#93;</span> and <span class="error">&#91;0, 0, 0, 0, 20, 50&#93;</span> (the level of the quota file is 4), then in the next start operation,<br/> "osd_it_acct_load" is called to resume the iteration, but it will change the block and index to<br/> <span class="error">&#91;1, 2, 3, 150, 200&#93;</span> and <span class="error">&#91;0, 0, 0, 30, 50&#93;</span>, that is because the last block storing the quota information will be used<br/> sequentially (will look for the first empty slot when a new quota needs to be inserted).</p> <p>2. there is a problem in "walk_tree_dqentry"</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>int walk_tree_dqentry(const struct lu_env *env, struct osd_object *obj, int type, uint blk, int depth, uint index, struct osd_it_quota *it) { dqbuf_t buf = getdqbuf(); loff_t ret; u32 *ref = (u32 *) buf; ENTRY; if (!buf) RETURN(-ENOMEM); ret = quota_read_blk(env, obj, type, blk, buf); if (ret &lt; 0) { CERROR("Can't read quota tree block %u.\n", blk); goto out_buf; } ret = 1; for (; index &lt;= 0xff &amp;&amp; ret &gt; 0; index++) { blk = le32_to_cpu(ref[index]); if (!blk) /* No reference */ continue; if (depth &lt; LUSTRE_DQTREEDEPTH - 1) ret = walk_tree_dqentry(env, obj, type, blk, depth + 1, 0, it); else ret = walk_block_dqentry(env, obj, type, blk, 0, it); &lt;--- here, if ret == 0, the "index++" in the "for" clause &lt;--- will still be called and cause one entry is skipped. } it-&gt;oiq_blk[depth + 1] = blk; it-&gt;oiq_index[depth] = index; out_buf: freedqbuf(buf); RETURN(ret); } </pre> </div></div> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31721" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31721</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> quota: fix issue of multiple call of seq start<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 24c196304b1e92279ee11b3eb9e0b4f49c11991c</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31721/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31721/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> quota: fix issue of multiple call of seq start<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 07fdba2244e748e1721232abc52617209a544b87</p> <p>Landed for 2.12</p> <p>We are seeing this again on 2.10.4<br/> <a href="https://testing.hpdd.intel.com/test_sets/e1084166-599e-11e8-b9d3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e1084166-599e-11e8-b9d3-52540065bddc</a></p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32567" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32567</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> quota: fix issue of multiple call of seq start<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: a9f70b640a443eb286848736f0ad1ed183c3a702</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32568" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32568</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> quota: fix issue of multiple call of seq start<br/> Project: fs/lustre-release<br/> Branch: b2_11<br/> Current Patch Set: 1<br/> Commit: 4822e044ca9296d4083b86c3667517fb0b2744c7</p> <p>Should not have been reopened for ticket affecting LTS branch</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32567/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32567/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8999" title="sanity-quota test_38: skipped id entries" class="issue-link" data-issue-key="LU-8999"><del>LU-8999</del></a> quota: fix issue of multiple call of seq start<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: cf796725fd39f0c807f60a86d2ddaa22e88c3a8c</p> Development Rank 1|hzz0av: Rank (Obsolete) 9223372036854775807 Severity