https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4180 Lustre <p>MDS hung with the following errors. There is some evidence that it happed right after setting a users quota to zero. Then after reboot/recover it hung again.</p> <p>&#8212; First set of errors messages<br/> &lt;3&gt;LustreError: 8304:0:(ldlm_lib.c:2467:target_handle_dqacq_callback()) dqacq/dqrel failed! (rc:-2)<br/> &lt;3&gt;LustreError: 8304:0:(ldlm_lib.c:2467:target_handle_dqacq_callback()) dqacq/dqrel failed! (rc:-2)<br/> &lt;3&gt;LustreError: 8304:0:(ldlm_lib.c:2467:target_handle_dqacq_callback()) dqacq/dqrel failed! (rc:-2)<br/> &lt;3&gt;LustreError: 8304:0:(ldlm_lib.c:2467:target_handle_dqacq_callback()) dqacq/dqrel failed! (rc:-2)<br/> &lt;3&gt;LustreError: 8304:0:(ldlm_lib.c:2467:target_handle_dqacq_callback()) dqacq/dqrel failed! (rc:-2)</p> <p>---- SOME STACK TRACES ----<br/> Stack traceback for pid 8262<br/> 0xffff883f8749cae0 8262 2 1 15 R 0xffff883f8749d180 ldlm_cb_26<br/> sp ip Function (args)<br/> ======================= &lt;nmi&gt;<br/> ======================= &lt;normal&gt;<br/> 0xffff883f9c8cf900 0xffffffff8151f04c _spin_lock+0x1c (0xffffffff8204c3a0)<br/> 0xffff883f9c8cf908 0xffffffff8135ce06 serial8250_console_write+0x156 (unknown, 0xffffffff81dc7012, unknown)<br/> 0xffff883f9c8cf958 0xffffffff81069745 __call_console_drivers+0x75 (unknown, unknown)<br/> 0xffff883f9c8cf988 0xffffffff810697aa _call_console_drivers+0x4a (unknown, unknown, unknown)<br/> 0xffff883f9c8cf9a8 0xffffffff81069d78 release_console_sem+0x108<br/> 0xffff883f9c8cf9e8 0xffffffff8106a478 vprintk+0x248 (unknown, unknown)<br/> 0xffff883f9c8cfa88 0xffffffff8151c151 printk+0x41<br/> 0xffff883f9c8cfae8 0xffffffffa04f8105 <span class="error">&#91;libcfs&#93;</span>cfs_print_to_console+0x75 (unknown, unknown, 0xffff883fd42b9182, unknown, unknown, 0xffffffffa07d7180)<br/> 0xffff883f9c8cfb18 0xffffffffa05038de <span class="error">&#91;libcfs&#93;</span>libcfs_debug_vmsg2+0x5de (0xffffffffa083de20, 0xffffffffa07f1c0e, 0xffff883f9c8cfc90, 0x0, 0x3, 0xffff8839c7b1cc00)<br/> 0xffff883f9c8cfc88 0xffffffffa0503ea1 <span class="error">&#91;libcfs&#93;</span>libcfs_debug_msg+0x41 (unknown, unknown, 0xfffffffe, 0xfffffffe, 0x3, 0xffff8839c7b1cc00)<br/> 0xffff883f9c8cfce8 0xffffffffa075e253 <span class="error">&#91;ptlrpc&#93;</span>target_handle_dqacq_callback+0x6e3 (0xffff881e0ef11400)<br/> 0xffff883f9c8cfd38 0xffffffffa076df87 <span class="error">&#91;ptlrpc&#93;</span>ldlm_callback_handler+0xa17 (0xffff881e0ef11400)<br/> 0xffff883f9c8cfdc8 0xffffffffa079fb4e <span class="error">&#91;ptlrpc&#93;</span>ptlrpc_main+0xc4e (unknown)<br/> 0xffff883f9c8cff48 0xffffffff8100c0ca child_rip+0xa (unknown, unknown)<br/> Enter &lt;q&gt; to end, &lt;cr&gt; to continue:</p> <p>&#8212; second set of errors after recover ----<br/> Lustre: MDS mdd_obd-nbp8-MDT0000: nbp8-OST001a_UUID now active, resetting orphans<br/> LustreError: 6186:0:(ldlm_lib.c:2467:target_handle_dqacq_callback()) dqacq/dqrel failed! (rc:-2)<br/> LustreError: 7771:0:(ldlm_lib.c:2467:target_handle_dqacq_callback()) dqacq/dqrel failed! (rc:-2)<br/> LustreError: 6185:0:(ldlm_lib.c:2467:target_handle_dqacq_callback()) dqacq/dqrel failed! (rc:-2)<br/> LustreError: 7798:0:(ldlm_lib.c:2467:target_handle_dqacq_callback()) dqacq/dqrel failed! (rc:-2)</p> <p>&#8212; Stack trace after reboot/recover -----<br/> Stack traceback for pid 8063<br/> 0xffff883fc6e92aa0 8063 2 1 8 R 0xffff883fc6e93140 qmaster_recovd<br/> sp ip Function (args)<br/> 0xffff883fc6ef7a18 0xffffffff8100c313 kdb_interrupt+0x13 (0xffffffffa05ada40, 0x0, 0x4e3a, 0x0, 0x0, 0xffff883f8a5bd940)<br/> 0xffff883fc6ef7a78 0xffffffff8151f04e _spin_lock+0x1e (0xffffffffa05ada40)<br/> 0xffff883fc6ef7aa8 0xffffffffa0582d76 <span class="error">&#91;lnet&#93;</span>LNetMDBind+0x1d6 (unknown, 0xffff883f9310c978, unknown, unknown, unknown, unknown, 0xffff883fc2e91e00, 0x100000130, 0xc6ef7ba0)<br/> 0xffff883fc6ef7af8 0xffffffffa0788478 <span class="error">&#91;ptlrpc&#93;</span>ptl_send_buf+0x128 (0xffff883f9310c978, 0xffff883fc2e91e00, unknown, unknown, 0xffff883f9310c980, 0xffff881e28d19b40, 0xffff883f0000001c, 0x52700f3927367, 0x1500000000)<br/> 0xffff883fc6ef7ba8 0xffffffffa078b774 <span class="error">&#91;ptlrpc&#93;</span>ptl_send_rpc+0x544 (0xffff883f9310c800, unknown)<br/> 0xffff883fc6ef7c68 0xffffffffa0781993 <span class="error">&#91;ptlrpc&#93;</span>ptlrpc_send_new_req+0x433 (0xffff883f9310c800)<br/> 0xffff883fc6ef7cd8 0xffffffffa078553a <span class="error">&#91;ptlrpc&#93;</span>ptlrpc_set_wait+0x6a (0xffff883f8bcdd8c0)<br/> 0xffff883fc6ef7d88 0xffffffffa0785cfc <span class="error">&#91;ptlrpc&#93;</span>ptlrpc_queue_wait+0x7c (0xffff883f9310c800)<br/> 0xffff883fc6ef7da8 0xffffffffa09c6011 <span class="error">&#91;lquota&#93;</span>client_quota_ctl+0x151 (unknown, unknown, 0xffff883fa47df5c0)<br/> 0xffff883fc6ef7de8 0xffffffffa09c6af7 <span class="error">&#91;lquota&#93;</span>lov_quota_ctl+0x427 (unknown, unknown, 0xffff883fa47df5c0)<br/> 0xffff883fc6ef7e48 0xffffffffa09cd9d9 <span class="error">&#91;lquota&#93;</span>obd_quotactl+0xd9 (0xffff881e9c387400, 0xffff883fa47df5c0)<br/> 0xffff883fc6ef7e78 0xffffffffa09d4ade <span class="error">&#91;lquota&#93;</span>qmaster_recovery_main+0x8ae (unknown)<br/> 0xffff883fc6ef7f48 0xffffffff8100c0ca child_rip+0xa (unknown, unknown)</p> <p><span class="error">&#91;3&#93;</span>kdb&gt; btp 7664<br/> Stack traceback for pid 7664<br/> 0xffff883fcdcad500 7664 2 1 27 R 0xffff883fcdcadba0 qmaster_recovd<br/> sp ip Function (args)<br/> 0xffff883fcdcaf998 0xffffffff8100c313 kdb_interrupt+0x13 (0xffffffffa05ada40, 0xffff883fc646e400, 0x4e3a, 0x0, 0x0, 0x130)<br/> 0xffff883fcdcaf9f8 0xffffffff8151f051 _spin_lock+0x21 (0xffffffffa05ada40)<br/> 0xffff883fcdcafa28 0xffffffffa05889f9 <span class="error">&#91;lnet&#93;</span>lnet_send+0x59 (0x500000a971b3c, 0xffff883fc646e400)<br/> 0xffff883fcdcafa98 0xffffffffa0589f04 <span class="error">&#91;lnet&#93;</span>LNetPut+0x2d4 (0x500000a971b3c, unknown, unknown, 0x500000a971b49, 0x3039, unknown, 0x52700f3927364, 0x100000000, 0x0)<br/> 0xffff883fcdcafaf8 0xffffffffa078852a <span class="error">&#91;ptlrpc&#93;</span>ptl_send_buf+0x1da (0xffff883fd7360978, 0xffff883f8e6a3c00, unknown, unknown, 0xffff883fd7360980, 0xffff881e28eca740, 0xffff883f0000001c, 0x52700f3927364, 0x1500000000)<br/> 0xffff883fcdcafba8 0xffffffffa078b774 <span class="error">&#91;ptlrpc&#93;</span>ptl_send_rpc+0x544 (0xffff883fd7360800, unknown)<br/> 0xffff883fcdcafc68 0xffffffffa0781993 <span class="error">&#91;ptlrpc&#93;</span>ptlrpc_send_new_req+0x433 (0xffff883fd7360800)<br/> 0xffff883fcdcafcd8 0xffffffffa078553a <span class="error">&#91;ptlrpc&#93;</span>ptlrpc_set_wait+0x6a (0xffff883f8a8b26c0)<br/> 0xffff883fcdcafd88 0xffffffffa0785cfc <span class="error">&#91;ptlrpc&#93;</span>ptlrpc_queue_wait+0x7c (0xffff883fd7360800)<br/> 0xffff883fcdcafda8 0xffffffffa09c6011 <span class="error">&#91;lquota&#93;</span>client_quota_ctl+0x151 (unknown, unknown, 0xffff883f8e8b3f40)<br/> 0xffff883fcdcafde8 0xffffffffa09c6af7 <span class="error">&#91;lquota&#93;</span>lov_quota_ctl+0x427 (unknown, unknown, 0xffff883f8e8b3f40)<br/> 0xffff883fcdcafe48 0xffffffffa09cd9d9 <span class="error">&#91;lquota&#93;</span>obd_quotactl+0xd9 (0xffff881e9c387400, 0xffff883f8e8b3f40)<br/> 0xffff883fcdcafe78 0xffffffffa09d4ade <span class="error">&#91;lquota&#93;</span>qmaster_recovery_main+0x8ae (unknown)<br/> 0xffff883fcdcaff48 0xffffffff8100c0ca child_rip+0xa (unknown, unknown)</p> LU-4180

ldlm_lib.c:2467:target_handle_dqacq_callback()) dqacq/dqrel failed! (rc:-2)

Bug Major Resolved Cannot Reproduce Niu Yawei Mahmoud Hanafi Tue, 29 Oct 2013 20:02:59 +0000 Thu, 7 Aug 2014 21:00:25 +0000 Thu, 7 Aug 2014 21:00:25 +0000 Lustre 2.1.5 0 4 <p>Niu</p> <p>Could you please comment?</p> <p>Thanks</p> <p>Peter</p> <p>This problem affects production systems.</p> <p>Setting console logging to 0 allowed the system to respond again.</p> <p>But /var/log/messages still scrolling these messages<br/> Oct 29 13:42:30 nbp8-mds1 kernel: LustreError: 6724:0:(ldlm_lib.c:2467:target_handle_dqacq_callback()) dqacq/dqrel failed! (rc:-2)<br/> Oct 29 13:42:30 nbp8-mds1 kernel: LustreError: 6724:0:(ldlm_lib.c:2467:target_handle_dqacq_callback()) dqacq/dqrel failed! (rc:-2)</p> <p>was able to get a D_TRACE and D_QUOTA on the MDS uploaded the files.</p> <p>The log shows dqacq_handler() returns -ENOENT on lustre_dqget(), but I don't see why we can't afford fake dquot in dqacq_handler(), it looks like a bug to me. See dqacq_handler():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> cfs_down_write(&amp;mds-&gt;mds_qonoff_sem); dquot = lustre_dqget(obd, info, qdata-&gt;qd_id, QDATA_IS_GRP(qdata), 0); ^^^ I think <span class="code-keyword">this</span> should be 1 <span class="code-keyword">if</span> (IS_ERR(dquot)) { cfs_up_write(&amp;mds-&gt;mds_qonoff_sem); GOTO(skip, rc = PTR_ERR(dquot)); } </pre> </div></div> <p>However, I didn't see why the quota_search_lqs() called before lustre_dqget() didn't return -ENOENT, I don't see where the lqs is inserted in the hash. Did you re-set limit for the user after reboot (beofre capture the log)?</p> <p>The limit was set BEFORE the log capture.</p> <p>This is sequence of events.<br/> 1. User id(2885) quota we set to zero. (lfs setquota -u pmoran -B 0 -b 0 -I 0 -i 0) this case the errors on the MDS and made it unresponsive.<br/> 2. The MDS was droped to kdb and stacktraces where taken<br/> 3. MDS was reboot<br/> 4. MDS Recovered and started printing the errors. <br/> 5. MDS was reboot gain but this time mdt was mounted with abort recovery<br/> 6. MDS started printing errors.<br/> 7. Console logging was turned off and the host became responsive to allow the debug dumps.<br/> 8. Quota was set for user id 2885 this stop the errors (lfs setquota -u pmoran -b 80000000000 -B 100000000000 -I 10000 -I 50000)</p> <p>I see, thanks Mahmoud. Looks something wrong happened when you clear the quota limit (lfs setquota -u pmoram -B 0 -b 0 -I 0 -i 0), that caused a partial global limit setting, and I believe quota recovery can't recover such failure (it can only recover the inconsistence of limits on master &amp; slave happened on dqacq/dqrel). </p> <p>I think re-set quota limit is the only way to recover this (just as what you did), could you also try to re-clear limit to see if things going well?</p> <p>In the new quota (2.4) architecture, such kind of partial global limit setting can be recovered automatically.</p> <p>re-clearing the quota didn't case the issue. But the MDS has been reboot a few time. We sill would like to understand why this happened and like to see it fixed.</p> <blockquote> <p>re-clearing the quota didn't case the issue.</p></blockquote> <p>Do you mean re-cleaning the quota limit didn't resolve the problem of "error message like ldlm_lib.c:2467:target_handle_dqacq_callback()) dqacq/dqrel failed! (rc:-2)" ?</p> <p>The error was resolved when I set the quota. But I wanted to reproduce the error by clearing/setting the quota to zero I was not able to reproduce it. At lease for now.</p> <blockquote> <p>The error was resolved when I set the quota. But I wanted to reproduce the error by clearing/setting the quota to zero I was not able to reproduce it. At lease for now.</p></blockquote> <p>To reproduce the problem:</p> <p>1. set 10M hardlimit for user xxx;<br/> 2. stripe a file owned by user xxx on OST1; (use command 'lfs setstripe -i 0 -c 1 $filename');<br/> 3. deactivate the OSC for OST1 on MDT to simulate connection problem between MDT and OST1; (use command 'lctl deactivate --device $oscdev deactivate' on MDS)<br/> 4. clear quota limit for user xxx;<br/> 5. reboot MDS;<br/> 6. write to the file as user xxx to see if the error messages showing up on MDS;</p> <p>Close no longer issue in 2.4.3</p> <p>Thanks Mahmoud</p> Development Rank 1|hzw78n: Rank (Obsolete) 11317 Severity