https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6382 Lustre <p>We have a quota problem on one of our OST.</p> <p>Here's the error logs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 11-0: lustre1-MDT0000-lwp-OST0008: Communicating with 10.225.8.3@o2ib, operation ldlm_enqueue failed with -3. LustreError: 12476:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed with -3, flags:0x9 qsd:lustre1-OST0008 qtype:grp id:10011 enforced:1 granted:1276244380 pending:0 waiting:128 req:1 usage:1276244415 qunit:0 qtune:0 edquot:0 LustreError: 12476:0:(qsd_handler.c:767:qsd_op_begin0()) $$$ ID isn't enforced on master, it probably due to a legeal race, if this message is showing up constantly, there could be some inconsistence between master &amp; slave, and quota reintegration needs be re-triggered. qsd:lustre1-OST0008 qtype:grp id:10011 enforced:1 granted:1276244380 pending:0 waiting:0 req:0 usage:1276244415 qunit:0 qtune:0 edquot:0 </pre> </div></div> <p>the errors occurs only on this OST. (and for that groupid only)</p> <p>We set the quotas with these commands:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs setquota -g $gid --block-softlimit 40t --block-hardlimit 40t /lustre1 lfs setquota -u $uid --inode-softlimit 1000000 --inode-hardlimit 1000000 /lustre1 </pre> </div></div> <p>and for the group 10011, we have disabled the quotas 1 or 2 days before the errors occur, using:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs setquota -g 10011 --block-softlimit 0 --block-hardlimit 0 /lustre1 </pre> </div></div> <p>What does mean "quota reintegration needs be re-triggered"? I guess it's to run an "lfs quotacheck" on the filesystem, right?</p> <p>Thanks<br/> JS</p> We are running lustre 2.5.3 on all our servers, with zfs 0.6.3 on the OSS and ldiskfs/ext4 on the MDS. (all 18 servers are running centos 6.5)<br/> The client nodes are running lustre 2.4.3 on centos 6.6<br/> LU-6382

quota : inconsistence between master & slave

Bug Minor Resolved Cannot Reproduce WC Triage JS Landry Wed, 18 Mar 2015 18:27:09 +0000 Wed, 8 Feb 2023 10:31:08 +0000 Sat, 9 Oct 2021 06:43:38 +0000 Lustre 2.4.3 Lustre 2.5.3 0 4 <p>The CEA hits this issue in production on a ClusterStor Lustre version (server side, 2.12.4...).<br/> Some users have quota id enforced on OSTs (slave, QSD), but not on MDT0000 (master, QMT). If the slave quota limits are exceeded (on OST), the clients fallback from BIO to sync IO:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-c"> <span class="code-keyword"><span class="code-object">int</span></span> vvp_io_write_commit(<span class="code-keyword">const</span> <span class="code-keyword">struct</span> lu_env *env, <span class="code-keyword">struct</span> cl_io *io) { ...... <span class="code-comment">/* out of quota, <span class="code-keyword">try</span> sync write */</span> if (rc == -EDQUOT &amp;&amp; !cl_io_is_mkwrite(io)) { <span class="code-keyword">struct</span> ll_inode_info *lli = ll_i2info(inode); rc = vvp_io_commit_sync(env, io, queue, </pre> </div></div> <p>This causes <b>a lot of small write IOs</b> from those user jobs on OSTs and increases quickly the load on OSS (raid6 parity calculations) and the disk usages (raid6 on rotational disk with no OST write cache). The overall filesystem was really slow.</p> <p>This issue has been resolved by forcing a quota reintegration on the OSS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param osd-ldiskfs.*.quota_slave.force_reint=1 </pre> </div></div> Related LU-4404 Development Epic/Theme Quota zfs Rank 1|hzx8un: Rank (Obsolete) 9223372036854775807 Severity