https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6629 Lustre <p>This issue was created by maloo for sarah_lw &lt;wei3.liu@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/bcb260f2-fe71-11e4-a865-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/bcb260f2-fe71-11e4-a865-5254006e85c2</a>.</p> <p>The sub-test test_bonnie failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test failed to respond and timed out </pre> </div></div> <p>this may be a dup of <a href="https://jira.whamcloud.com/browse/LU-4875" title="We have 2 OSS server in HA and two MDS in HA , On each OSS 12 OSTs are mounted per OSS with faiover. OSS servers get reboots while working" class="issue-link" data-issue-key="LU-4875"><del>LU-4875</del></a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>04:45:24:Lustre: DEBUG MARKER: == sanity-benchmark test bonnie: bonnie++ == 04:14:01 (1431922441) 04:45:24:Lustre: DEBUG MARKER: /usr/sbin/lctl mark min OST has 1969152kB available, using 3844624kB file size 04:45:24:Lustre: DEBUG MARKER: min OST has 1969152kB available, using 3844624kB file size 04:45:24:LNet: Service thread pid 3517 completed after 77.68s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). 04:45:24:LNet: Skipped 8 previous similar messages 04:45:24:LNet: Service thread pid 3556 completed after 84.32s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). 04:45:24:LustreError: 7365:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed with -22, flags:0x4 qsd:lustre-OST0004 qtype:grp id:500 enforced:1 granted:1048576 pending:0 waiting:0 req:1 usage:0 qunit:0 qtune:0 edquot:0 04:45:24:LustreError: 7365:0:(qsd_handler.c:340:qsd_req_completion()) Skipped 12 previous similar messages 04:45:24:LustreError: 7364:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed with -22, flags:0x4 qsd:lustre-OST0004 qtype:grp id:500 enforced:1 granted:1048576 pending:0 waiting:0 req:1 usage:0 qunit:0 qtune:0 edquot:0 04:45:24:LustreError: 7364:0:(qsd_handler.c:340:qsd_req_completion()) Skipped 11 previous similar messages 04:45:24:LustreError: 7364:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed with -22, flags:0x4 qsd:lustre-OST0004 qtype:grp id:500 enforced:1 granted:1048576 pending:0 waiting:0 req:1 usage:0 qunit:0 qtune:0 edquot:0 04:45:24:LustreError: 7364:0:(qsd_handler.c:340:qsd_req_completion()) Skipped 11 previous similar messages 04:45:24:LNet: Service thread pid 29001 completed after 45.78s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). 04:45:24:LNet: Skipped 11 previous similar messages 05:14:26:********** Timeout by autotest system ********** </pre> </div></div> lustre-master build #3029 LU-6629

sanity-benchmark test_bonnie: DQACQ failed with -22

Bug Blocker Resolved Fixed Niu Yawei Maloo zfs Thu, 21 May 2015 20:32:01 +0000 Mon, 30 Jan 2017 19:21:46 +0000 Fri, 18 Nov 2016 04:18:19 +0000 Lustre 2.8.0 0 10 <p>I also see in the MDS logs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>04:15:58:LustreError: 14209:0:(qmt_handler.c:420:qmt_dqacq0()) $$$ Release too much! uuid:lustre-MDT0000-lwp-OST0004_UUID release:1048576 granted:0, total:4194304 qmt:lustre-QMT0000 pool:0-dt id:500 enforced:1 hard:8533324 soft:8126976 granted:4194304 time:0 qunit:1048576 edquot:0 may_rel:0 revoke:0 </pre> </div></div> <p>Looks two slaves (OST4 &amp; OST5) are not synced with master, I can't see from log how this happened, but I think this should not be the cause of "too many service threads, or there were not enough hardware resources".</p> <p>There was a defect could leads to quota slave reconnect without invalidate global locks, that could result in the quota slave &amp; master not synced at the end. I think this has been fixed by 4f53536d002c13886210b672b657795baa067144</p> <p>If the error message of "04:15:58:LustreError: 14209:0:(qmt_handler.c:420:qmt_dqacq0()) $$$ Release too much! " were not seen on master anymore, I think we can close this ticket.</p> <p>This should have been fixed by following changes in the commit of 4f53536d002c13886210b672b657795baa067144 :</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">+ /* Note: lw_client is needed in MDS-MDS failover during update log + * processing, so we needs to allow lw_client to be connected at + * anytime, instead of only the initial connection */ + lw_client = (data-&gt;ocd_connect_flags &amp; OBD_CONNECT_LIGHTWEIGHT) != 0; + <span class="code-keyword">if</span> (lustre_msg_get_op_flags(req-&gt;rq_reqmsg) &amp; MSG_CONNECT_INITIAL) { mds_conn = (data-&gt;ocd_connect_flags &amp; OBD_CONNECT_MDS) != 0; - lw_client = (data-&gt;ocd_connect_flags &amp; - OBD_CONNECT_LIGHTWEIGHT) != 0; </pre> </div></div> <p>Patch landed.</p> Related Development Rank 1|hzxdun: Rank (Obsolete) 9223372036854775807 Severity