https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10368 Lustre <p>Hi,</p> <p> </p> <p>We are seeing quota problems with 2.10.1 where, from time to time, group quotas are generating EDQUOT (users are actually reporting the problem) while there is room left. Some OSTs are seen as full as shown below and the rebalancing doesn't seem to work:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[root@oak-rbh01 ~]# lfs quota -v -g oak_p-cvmed /oak Disk quotas <span class="code-keyword">for</span> grp oak_p-cvmed (gid 3683): Filesystem kbytes quota limit grace files quota limit grace /oak 96389875036 120000000000 120000000000 - 6955998 18000000 18000000 - oak-MDT0000_UUID 3248820 - 0 - 6955998 - 8388608 - oak-OST0000_UUID 1196660392 - 1342177280 - - - - - oak-OST0001_UUID 1908235316 - 2147483648 - - - - - oak-OST0002_UUID 1225832336 - 1342177280 - - - - - oak-OST0003_UUID 1327591244 - 1342177280 - - - - - oak-OST0004_UUID 1965895968 - 2147483648 - - - - - oak-OST0005_UUID 1159391612 - 1342177280 - - - - - oak-OST0006_UUID 1635255040 - 1879048192 - - - - - oak-OST0007_UUID 1818596964 - 1879048192 - - - - - oak-OST0008_UUID 1872031764 - 1879048192 - - - - - oak-OST0009_UUID 2061279604 - 2147483648 - - - - - oak-OST000a_UUID 1445543488 - 1610612736 - - - - - oak-OST000b_UUID 1875314700 - 1879048192 - - - - - oak-OST000c_UUID 1301881412 - 1342177280 - - - - - oak-OST000d_UUID 1766688092 - 1879048192 - - - - - oak-OST000e_UUID 2005981712 - 2147483648 - - - - - oak-OST000f_UUID 1491138396 - 1610612736 - - - - - oak-OST0010_UUID 1292096088 - 1342177280 - - - - - oak-OST0011_UUID 1222866272 - 1342177280 - - - - - oak-OST0012_UUID 1312869104 - 1342177280 - - - - - oak-OST0013_UUID 1185445504 - 1342177280 - - - - - oak-OST0014_UUID 1315544800 - 1342177280 - - - - - oak-OST0015_UUID 2025717256 - 2147483648 - - - - - oak-OST0016_UUID 1817010800 - 1879048192 - - - - - oak-OST0017_UUID 1699092560 - 1879048192 - - - - - oak-OST0018_UUID 1921966992 - 2147483648 - - - - - oak-OST0019_UUID 1752975104 - 1879048192 - - - - - oak-OST001a_UUID 2022449576 - 2147483648 - - - - - oak-OST001b_UUID 1476019956 - 1610612736 - - - - - oak-OST001c_UUID 2002420900 - 2147483648 - - - - - oak-OST001d_UUID 1175776272 - 1342177280 - - - - - oak-OST001e_UUID 1522667428 - 1610612736 - - - - - oak-OST001f_UUID 1698940868 - 1879048192 - - - - - oak-OST0020_UUID 1418438600 - 1610612736 - - - - - oak-OST0021_UUID 1848558676 - 1879048192 - - - - - oak-OST0022_UUID 1567670312 - 1610612736 - - - - - oak-OST0023_UUID 1755882404 - 1879048192 - - - - - oak-OST0024_UUID 1725770704 - 1879048192 - - - - - oak-OST0025_UUID 2021496552 - 2147483648 - - - - - oak-OST0026_UUID 2340218652 - 2415919104 - - - - - oak-OST0027_UUID 2078849960 - 2147483648 - - - - - oak-OST0028_UUID 2401223300 - 2415919104 - - - - - oak-OST0029_UUID 2255153880 - 2415919104 - - - - - oak-OST002a_UUID 2479360100 - 2684354560 - - - - - oak-OST002b_UUID 1956889380 - 2147483648 - - - - - oak-OST002c_UUID 2336034612 - 2415919104 - - - - - oak-OST002d_UUID 1897045500 - 2147483648 - - - - - oak-OST002e_UUID 2069066412 - 2147483648 - - - - - oak-OST002f_UUID 2668099124 - 2684354560 - - - - - oak-OST0030_UUID 302970856 - 536870912 - - - - - oak-OST0031_UUID 425767268 - 536870912 - - - - - oak-OST0032_UUID 554265344 - 805306368 - - - - - oak-OST0033_UUID 616158116 - 805306368 - - - - - oak-OST0034_UUID 523406904 - 536870912 - - - - - oak-OST0035_UUID 832949332 - 1073741824 - - - - - oak-OST0036_UUID 431649588 - 536870912 - - - - - oak-OST0037_UUID 335297304 - 536870912 - - - - - oak-OST0038_UUID 768953372 - 805306368 - - - - - oak-OST0039_UUID 589398720 - 805306368 - - - - - oak-OST003a_UUID 822149664 - 1073741824 - - - - - oak-OST003b_UUID 246038976 - 268435456 - - - - - oak-OST003c_UUID 1002757608 - 1073741824 - - - - - oak-OST003d_UUID 655190956 - 805306368 - - - - - oak-OST003e_UUID 464755608* - 464755608 - - - - - oak-OST003f_UUID 265537376 - 268435456 - - - - - oak-OST0040_UUID 380491764 - 536870912 - - - - - oak-OST0041_UUID 628194908 - 805306368 - - - - - oak-OST0042_UUID 220394524 - 268435456 - - - - - oak-OST0043_UUID 388284936 - 536870912 - - - - - oak-OST0044_UUID 429979492 - 536870912 - - - - - oak-OST0045_UUID 276764380 - 536870912 - - - - - oak-OST0046_UUID 346999308 - 536870912 - - - - - oak-OST0047_UUID 408032656 - 536870912 - - - - - oak-OST0048_UUID 17760704 - 268435456 - - - - - oak-OST0049_UUID 11267016 - 268435456 - - - - - oak-OST004a_UUID 12111344 - 268435456 - - - - - oak-OST004b_UUID 8750240 - 268435456 - - - - - oak-OST004c_UUID 31825656 - 268435456 - - - - - oak-OST004d_UUID 67586608 - 268435456 - - - - - Total allocated inode limit: 8388608, total allocated block limit: 106765196184 </pre> </div></div> <p> In some cases I was able to disable/enable group q uota to fix the problem but in this particular, I can't find a way to force a refresh. Any idea would be welcome? This has a pretty important impact on some groups.</p> <p> </p> <p>Thanks!</p> <p>Stephane</p> <p> </p> CentOS 7.4 LU-10368

disk quota OST rebalancing issues

Bug Critical Resolved Fixed Hongchao Zhang Stephane Thiell Mon, 11 Dec 2017 23:19:35 +0000 Thu, 3 May 2018 20:29:29 +0000 Thu, 19 Apr 2018 13:08:10 +0000 Lustre 2.10.1 Lustre 2.12.0 Lustre 2.10.4 0 5 <p>Thanks Stephane</p> <p>Just upgraded the servers to 2.10.2 and we'll see how it goes (right now I can't reproduce the issue).</p> <p>I'm also noticing the following lquota log messages (also reported in -<del><a href="https://jira.whamcloud.com/browse/LU-9790" title="ost-pools test_23a: Quota failed to enable in ZFS 0.7.0-rc5" class="issue-link" data-issue-key="LU-9790"><del>LU-9790</del></a></del>-):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">00040000:02020000:27.0F:1513208262.246875:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST003c: can<span class="code-quote">'t enable quota enforcement since space accounting isn'</span>t functional. Please run tunefs.lustre --quota on an unmounted filesystem <span class="code-keyword">if</span> not done already 00040000:02020000:27.0:1513208262.272624:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST0038: can<span class="code-quote">'t enable quota enforcement since space accounting isn'</span>t functional. Please run tunefs.lustre --quota on an unmounted filesystem <span class="code-keyword">if</span> not done already 00040000:02020000:27.0:1513208262.272627:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST0034: can<span class="code-quote">'t enable quota enforcement since space accounting isn'</span>t functional. Please run tunefs.lustre --quota on an unmounted filesystem <span class="code-keyword">if</span> not done already 00040000:02020000:27.0:1513208262.272628:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST0042: can<span class="code-quote">'t enable quota enforcement since space accounting isn'</span>t functional. Please run tunefs.lustre --quota on an unmounted filesystem <span class="code-keyword">if</span> not done already 00040000:02020000:27.0:1513208262.272629:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST0046: can<span class="code-quote">'t enable quota enforcement since space accounting isn'</span>t functional. Please run tunefs.lustre --quota on an unmounted filesystem <span class="code-keyword">if</span> not done already 00040000:02020000:27.0:1513208262.272630:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST003a: can<span class="code-quote">'t enable quota enforcement since space accounting isn'</span>t functional. Please run tunefs.lustre --quota on an unmounted filesystem <span class="code-keyword">if</span> not done already 00040000:02020000:27.0:1513208262.272631:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST004c: can<span class="code-quote">'t enable quota enforcement since space accounting isn'</span>t functional. Please run tunefs.lustre --quota on an unmounted filesystem <span class="code-keyword">if</span> not done already 00040000:02020000:27.0:1513208262.272632:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST004a: can<span class="code-quote">'t enable quota enforcement since space accounting isn'</span>t functional. Please run tunefs.lustre --quota on an unmounted filesystem <span class="code-keyword">if</span> not done already </pre> </div></div> <p>Our setup is: space acct on ug (no project quota, ie. no ldiskfs flag) and quota only enabled for groups (g):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">target name: oak-OST004c pool ID: 0 type: dt quota enabled: g conn to master: setup space acct: ug user uptodate: glb[0],slv[0],reint[0] group uptodate: glb[1],slv[1],reint[0] project uptodate: glb[0],slv[0],reint[0] </pre> </div></div> <p> </p> <p> </p> <p>Best,</p> <p>Stephane</p> <p><a href="https://testing.hpdd.intel.com/test_sets/9488b58c-f1b6-11e7-a169-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9488b58c-f1b6-11e7-a169-52540065bddc</a></p> <p>see similar message 'can't enable quota enforcement'</p> <p>Hongchao</p> <p>Can you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>the bug in <a href="https://testing.hpdd.intel.com/test_sets/9488b58c-f1b6-11e7-a169-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9488b58c-f1b6-11e7-a169-52540065bddc</a> is not the same issue,<br/> it's caused by the delayed response from MDT and client reconnected to MDT, which cause the MDS_QUOTACTL<br/> request in "mdc_quotactl" to fail for it is marked as "no resend".</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static int mdc_quotactl(struct obd_device *unused, struct obd_export *exp, struct obd_quotactl *oqctl) { struct ptlrpc_request *req; struct obd_quotactl *oqc; int rc; ENTRY; req = ptlrpc_request_alloc_pack(class_exp2cliimp(exp), &amp;RQF_MDS_QUOTACTL, LUSTRE_MDS_VERSION, MDS_QUOTACTL); if (req == NULL) RETURN(-ENOMEM); oqc = req_capsule_client_get(&amp;req-&gt;rq_pill, &amp;RMF_OBD_QUOTACTL); *oqc = *oqctl; ptlrpc_request_set_replen(req); ptlrpc_at_set_req_timeout(req); req-&gt;rq_no_resend = 1; &lt;--- here rc = ptlrpc_queue_wait(req); if (rc) CERROR("ptlrpc_queue_wait failed, rc: %d\n", rc); if (req-&gt;rq_repmsg &amp;&amp; (oqc = req_capsule_server_get(&amp;req-&gt;rq_pill, &amp;RMF_OBD_QUOTACTL))) { *oqctl = *oqc; } else if (!rc) { CERROR ("Can't unpack obd_quotactl\n"); rc = -EPROTO; } ptlrpc_req_finished(req); RETURN(rc); } </pre> </div></div> <p>the corresponding logs are</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 8535.569851] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ================================================ 22:08:14 (1515103694) [ 9410.140055] Lustre: 19972:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1515103967/real 1515103967] req@ffff88006518cc00 x1588697858744704/t0(0) o48-&gt;lustre-MDT0000-mdc-ffff88006b606800@10.9.4.248@tcp:12/10 lens 336/336 e 21 to 1 dl 1515104568 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9410.148998] Lustre: lustre-MDT0000-mdc-ffff88006b606800: Connection to lustre-MDT0000 (at 10.9.4.248@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9410.278389] Lustre: lustre-MDT0000-mdc-ffff88006b606800: Connection restored to 10.9.4.248@tcp (at 10.9.4.248@tcp) [ 9410.281474] LustreError: 19972:0:(mdc_request.c:1840:mdc_quotactl()) ptlrpc_queue_wait failed, rc: -107 [ 9410.462846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity-quota test_8: @@@@@@ FAIL: clear quota for [type:-u name:quota_usr] failed </pre> </div></div> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31773" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31773</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10368" title="disk quota OST rebalancing issues" class="issue-link" data-issue-key="LU-10368"><del>LU-10368</del></a> mdc: resend quotactl if needed<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 30466952d14f2e891f3a1bdd29103ae578f00413</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31773/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31773/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10368" title="disk quota OST rebalancing issues" class="issue-link" data-issue-key="LU-10368"><del>LU-10368</del></a> mdc: resend quotactl if needed<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d511918e8eb725abba2561cc493e30651a89ac27</p> <p>Stephane</p> <p>I'm not sure whether it is easy for you to test out this fix and confirm whether it resolves the issue, but if you do have a way then that would be much appreciated. For now I'll mark this ticket as resolved - "innocent until proven guilty" <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> - and we'll queue it up for a future 2.10.x release.</p> <p>Peter</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32075" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32075</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10368" title="disk quota OST rebalancing issues" class="issue-link" data-issue-key="LU-10368"><del>LU-10368</del></a> mdc: resend quotactl if needed<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: e90b1a190b38db9573a5284ca53efbff612d0972</p> <p>Hi Peter,</p> <p>I'll try the patch at the first opportunity after LUG.</p> <p>Please also note that we haven't noticed any new occurence of this issue in 2.10.3 so far.</p> <p>Thanks!</p> <p>Stephane</p> <p> </p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/32075/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32075/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10368" title="disk quota OST rebalancing issues" class="issue-link" data-issue-key="LU-10368"><del>LU-10368</del></a> mdc: resend quotactl if needed<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: c2c8a3f6dec17144f317aab409f48b862d9aa1b1</p> Related Development Rank 1|hzzp67: Rank (Obsolete) 9223372036854775807 Severity