https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4857 Lustre <p>hi,</p> <p>i'm seeing following messages from 2.4.2 OSS every few minutes:</p> <p>2014-04-02T16:32:48+11:00 lemming17 kernel: LustreError: 17701:0:(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -22, flags:0x4 qsd:short-OST011f qtype:grp id:6644 enforced:1 granted:1048576 pending:0 waiting:0 req:1 usage:0 qunit:0 qtune:0 edquot:0<br/> 2014-04-02T16:32:48+11:00 lemming17 kernel: LustreError: 17701:0:(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -22, flags:0x4 qsd:short-OST011f qtype:grp id:6644 enforced:1 granted:1048576 pending:0 waiting:0 req:1 usage:0 qunit:0 qtune:0 edquot:0<br/> 2014-04-02T16:31:50+11:00 lemming27 kernel: LustreError: 21284:0:qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -22, flags:0x4 qsd:short-OST0115 qtype:grp id:6644 enforced:1 granted:1048576 pending:0 waiting:0 req:1 usage:0 qunit:0 qtune:0 edquot:0<br/> 2014-04-02T16:31:50+11:00 lemming27 kernel: LustreError: 21284:0:(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -22, flags:0x4 qsd:short-OST0115 qtype:grp id:6644 enforced:1 granted:1048576 pending:0 waiting:0 req:1 usage:0 qunit:0 qtune:0 edquot:0</p> <p>surprisingly, the errors are spewed only from two OSSes out of the lot and only for specific OSTs. </p> <p>at the same time MDS is throwing following:</p> <p>2014-04-02T16:32:36+11:00 gerbil5 kernel: LustreError: 17470:0:(qmt_handler.c:431:qmt_dqacq0()) $$$ Release too much! uuid:short-MDT0000-lwp-OST011f_UUID release:1048576 granted:0, total:354880716 qmt:short-QMT0000 pool:0-dt id:6644 enforced:1 hard:2516582400 soft:12582 91200 granted:354880716 time:0 qunit:1048576 edquot:0 may_rel:0 revoke:0<br/> 2014-04-02T16:32:36+11:00 gerbil5 kernel: LustreError: 17470:0:(qmt_handler.c:431:qmt_dqacq0()) $$$ Release too much! uuid:short-MDT000 0-lwp-OST011f_UUID release:1048576 granted:0, total:354880716 qmt:short-QMT0000 pool:0-dt id:6644 enforced:1 hard:2516582400 soft:12582 91200 granted:354880716 time:0 qunit:1048576 edquot:0 may_rel:0 revoke:0<br/> 2014-04-02T16:32:39+11:00 gerbil5 kernel: LustreError: 4733:0:(qmt_handler.c:431:qmt_dqacq0()) $$$ Release too much! uuid:short-MDT0000 -lwp-OST0115_UUID release:1048576 granted:0, total:354880716 qmt:short-QMT0000 pool:0-dt id:6644 enforced:1 hard:2516582400 soft:1258291200 granted:354880716 time:0 qunit:1048576 edquot:0 may_rel:0 revoke:0<br/> 2014-04-02T16:32:39+11:00 gerbil5 kernel: LustreError: 4733:0:(qmt_handler.c:431:qmt_dqacq0()) $$$ Release too much! uuid:short-MDT0000-lwp-OST0115_UUID release:1048576 granted:0, total:354880716 qmt:short-QMT0000 pool:0-dt id:6644 enforced:1 hard:2516582400 soft:1258291200 granted:354880716 time:0 qunit:1048576 edquot:0 may_rel:0 revoke:0</p> <p>these errors are seen after all the servers have been rebooted afresh as part of maintenance cycle (some other LBUGs were fixed).</p> <p>any pointers what could be causing it?</p> CentOS 6.4 / Kernel 2.6.32-358.18.1.el6_lustre.x86_64 LU-4857

(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -22

Bug Minor Resolved Duplicate Niu Yawei javed shaikh Thu, 3 Apr 2014 06:51:14 +0000 Thu, 23 Jun 2016 08:16:02 +0000 Thu, 23 Jun 2016 08:16:02 +0000 Lustre 2.4.2 0 4 <p>This looks like the quota slave index files on master and slave are not synchronized somehow.</p> <p>1. Did the error message show up without a stop, or it disappeared after a while?<br/> 2. Check the quota_slave.info proc file for the OST with erros. (cat /proc/fs/lustre/osd-ldiskfs/$OSTNAME/quota_slave/info)<br/> 3. Try to trigger reintegration to see if it can solve the problem. (ehco 1 &gt; /proc/fs/lustre/osd-ldiskfs/$OSTNAME/quota_slave/force_reint)</p> <p>If force reintegration can't resolve the problem, could you collect lustre log with D_QUOTA and D_TRACE enabled? (execute following instructions on the OSS to collect log)</p> <ul class="alternate" type="square"> <li>lctl clear</li> <li>lctl set_param debug=+trace; lctl set_param debug=+quota</li> <li>lctl debug_deamon start $TMPFILE 500</li> <li>lctl mark "============= start reint ================="</li> <li>ehco 1 &gt; /proc/fs/lustre/osd-ldiskfs/$OSTNAME/quota_slave/force_reint</li> <li>wait for reintegration done. (when the "user uptodate" and "group uptodate" in /proc/fs/lustre/osd-ldiskfs/$OSTNAME/quota_slave/info changes to "glb<span class="error">&#91;1&#93;</span>,slv<span class="error">&#91;1&#93;</span>,reint<span class="error">&#91;0&#93;</span>")</li> <li>lctl debug_daemon stop</li> <li>lctl debug_file $TMPFILE $LOGFILE</li> <li>attach log file in the ticket.</li> </ul> <p>1. error msgs rolled every 10mins<br/> 3. triggered reintergation and it seems to have stopped both OSS and MDS errors<br/> 2. this is after reintegration:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ cat /proc/fs/lustre/osd-ldiskfs/<span class="code-object">short</span>-OST0115/quota_slave/info target name: <span class="code-object">short</span>-OST0115 pool ID: 0 type: dt quota enabled: g conn to master: setup space acct: ug user uptodate: glb[0],slv[0],reint[0] group uptodate: glb[1],slv[1],reint[0] </pre> </div></div> <p>i'll observe for a day and then get back. thanks.</p> <p>update.<br/> it did stop on the above OSS/OSTs. but started on other OSS/OSTs in last two days. forced reintegration stops it.<br/> so guess, need to keep forcing as cases crop up.</p> <p>this ticket can be closed. thanks.</p> <p>Hi, Javed, is there any error messages in the MDT or OST before this start? I'd like to see why the reintegration failed.</p> <p>just checked for one of affected OSTs. no error msgs. only msgs related to client evictions, orphan obj deletions.</p> <p>just got this from another OSS:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">2014-04-10T17:56:09+10:00 lemming2 kernel: Lustre: <span class="code-object">short</span>-OST008e: haven<span class="code-quote">'t heard from client 4ab84a25-da95-49af-a512-e406eb4af84f (at 10.9.3.2@o2ib3) in 227 seconds. I think it'</span>s dead, and I am evicting it. exp ffff8820371ff400, cur 1397116569 expire 1397116419 last 1397116342 2014-04-10T17:56:09+10:00 lemming2 kernel: Lustre: Skipped 20 previous similar messages 2014-04-11T09:20:49+10:00 lemming2 kernel: LustreError: 25971:0:(ldlm_resource.c:1165:ldlm_resource_get()) <span class="code-object">short</span>-OST008e: lvbo_init failed <span class="code-keyword">for</span> resource 0xb65a43:0x0: rc = -2 2014-04-11T10:08:10+10:00 lemming2 kernel: LustreError: 25593:0:(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -22, flags:0x4 qsd:<span class="code-object">short</span>-OST008e qtype:grp id:4093 enforced:1 granted:65536 pending:0 waiting:0 req:1 usage:0 qunit:0 qtune:0 edquot:0 </pre> </div></div> <p>though there's one more OSS where the DQACQ error occurred on OST without prior errors on that OST.</p> <p>dup of <a href="https://jira.whamcloud.com/browse/LU-6629" title="sanity-benchmark test_bonnie: DQACQ failed with -22" class="issue-link" data-issue-key="LU-6629"><del>LU-6629</del></a></p> Development Rank 1|hzwj67: Rank (Obsolete) 13395 Severity