https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4249 Lustre <p>OSS crashed with Projection Fault at lqe64_hash_keycmp</p> <p>See attached files. <br/> Looks like we are crashing here</p> <p>libcfs/libcfs/hash.c<br/> static cfs_hlist_node_t *<br/> cfs_hash_bd_lookup_intent(cfs_hash_t *hs, cfs_hash_bd_t *bd, <br/> .<br/> .<br/> .<br/> cfs_hlist_for_each(ehnode, hhead) { &lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt; CRASH <br/> if (!cfs_hash_keycmp(hs, key, ehnode))<br/> continue;</p> <p>cfs_hlist_for_each defined as hlist_for_each but hlist_for_each doesn't seem to be defined any where.</p> <p>See attached backtrace info.</p> our source tree is at <a href="https://github.com/jlan/lustre-nas">https://github.com/jlan/lustre-nas</a><br/> Server running 2.4.0-4nasS LU-4249

exception RIP: lqe64_hash_keycmp+12

Bug Major Resolved Fixed Niu Yawei Mahmoud Hanafi Wed, 13 Nov 2013 22:19:53 +0000 Thu, 26 Feb 2015 22:08:48 +0000 Fri, 8 Aug 2014 13:13:04 +0000 Lustre 2.4.0 Lustre 2.5.3 0 16 <p>hlist_for_each() is a kernel function. Mahmoud, do you know if this happened when shutding down OST? Thanks.</p> <p>This was not during shutdown</p> <p>We hit this bug again today. Is there any progress coming up with a cause/fix?</p> <p>Mahmoud, could you please attach the console log from before the crash and the oops message itself.</p> <p>Here 1 Hour before the oops and the oops message itself.<br/> <span class="error">&#91;-- MARK -- Fri Nov 15 11:00:00 2013&#93;</span>^M<br/> LustreError: 13805:0:(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x2 qsd:nbp7-OST004e qtype:usr id:12157 enforced:1 granted:3639632 pending:0 waiting:0 req:1 usage:3449480 qunit:1048576 qtune:262144 edquot:0^M<br/> LustreError: 13805:0:(qsd_handler.c:344:qsd_req_completion()) Skipped 52 previous similar messages^M<br/> LustreError: 13891:0:(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x2 qsd:nbp7-OST002a qtype:usr id:12157 enforced:1 granted:4194304 pending:0 waiting:0 req:1 usage:4009436 qunit:1048576 qtune:262144 edquot:0^M<br/> LustreError: 13891:0:(qsd_handler.c:344:qsd_req_completion()) Skipped 58 previous similar messages^M<br/> LustreError: 14039:0:(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x2 qsd:nbp7-OST002e qtype:usr id:12157 enforced:1 granted:4157724 pending:0 waiting:0 req:1 usage:3963432 qunit:1048576 qtune:262144 edquot:0^M<br/> LustreError: 14039:0:(qsd_handler.c:344:qsd_req_completion()) Skipped 48 previous similar messages^M<br/> Lustre: nbp7-OST004e: haven't heard from client fefdd2bf-1bc8-70b6-3aeb-de9d92f342d8 (at 10.151.1.109@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88155de95400, cur 1384543303 expire 1384543153 last 1384543076^M<br/> Lustre: Skipped 10 previous similar messages^M<br/> Lustre: nbp7-OST0042: haven't heard from client fefdd2bf-1bc8-70b6-3aeb-de9d92f342d8 (at 10.151.1.109@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881a0cb53000, cur 1384543303 expire 1384543153 last 1384543076^M<br/> LNet: 2974:0:(o2iblnd_cb.c:2348:kiblnd_passive_connect()) Conn stale 10.151.17.83@o2ib <span class="error">&#91;old ver: 12, new ver: 12&#93;</span>^M<br/> LustreError: 14039:0:(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x2 qsd:nbp7-OST002e qtype:usr id:12157 enforced:1 granted:4157724 pending:0 waiting:0 req:1 usage:3963432 qunit:1048576 qtune:262144 edquot:0^M<br/> LustreError: 14039:0:(qsd_handler.c:344:qsd_req_completion()) Skipped 562 previous similar messages^M<br/> general protection fault: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP ^M<br/> last sysfs file: /sys/devices/pci0000:80/0000:80:03.0/0000:86:00.0/host15/rport-15:0-0/target15:0:0/15:0:0:39/state^M<br/> ^M<br/> Entering kdb (current=0xffff8810259ed500, pid 306) on processor 19 Oops: (null)^M<br/> due to oops @ 0xffffffffa0c72f2c^M<br/> r15 = 0x5a5a5a5a5a5a5a5a r14 = 0x0000000000000000 ^M<br/> r13 = 0xffff880e9509ef28 r12 = 0xffff8810259ef9d0 ^M<br/> bp = 0xffff8810259ef940 bx = 0xffff881f55c5f900 ^M<br/> r11 = 0x6000000000000000 r10 = 0x0000000000001e70 ^M<br/> r9 = 0x000000000000e56f r8 = 0x0000000000000003 ^M<br/> ax = 0x0000000000002f7d cx = 0x0000000000000000 ^M<br/> dx = 0x0000000000000000 si = 0x5a5a5a5a5a5a5a5a ^M<br/> di = 0xffff880e9509ef28 orig_ax = 0xffffffffffffffff ^M<br/> ip = 0xffffffffa0c72f2c cs = 0x0000000000000010 ^M<br/> flags = 0x0000000000010206 sp = 0xffff8810259ef930 ^M<br/> ss = 0x0000000000000018 &amp;regs = 0xffff8810259ef898^M<br/> <span class="error">&#91;19&#93;</span>kdb&gt; <span class="error">&#91;-- MARK -- Fri Nov 15 12:00:00 2013&#93;</span>^M<br/> <span class="error">&#91;-- root@localhost attached -- Fri Nov 15 12:01:54 2013&#93;</span>^M<br/> <span class="error">&#91;-- root@localhost detached -- Fri Nov 15 12:08:44 2013&#93;</span>^M<br/> <span class="error">&#91;-- root@localhost attached -- Fri Nov 15 12:10:07 2013&#93;</span>^M</p> <p>Looks like the lqe is freed unexpectedly, but I didn't see from the code how it will be freed in such case. Anyway, the message:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 13891:0:(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x2 qsd:nbp7-OST002a qtype:usr id:12157 enforced:1 granted:4194304 pending:0 waiting:0 req:1 usage:4009436 qunit:1048576 qtune:262144 edquot:0^M </pre> </div></div> <p>indicating something was wrong. If such error message can be reproduced, could you reproduce it and capture the logs on both OST and MDT with D_QUOTA &amp; D_TRACE enabled? I'd like to figure out why the acquire request always failed with -ENOLCK.</p> <p>Uploaded debug logs:</p> <p> <a href="ftp://ftp.whamcloud.com/uploads/LU-4249.debuglogs.tgz" class="external-link" target="_blank" rel="nofollow noopener">ftp://ftp.whamcloud.com/uploads/LU-4249.debuglogs.tgz</a></p> <p>The numerous ENOLCK errors is possibly caused by a race:</p> <ul class="alternate" type="square"> <li>The per-ID quota lock is being canceled on quota slave, and the corresponding lqe_lockh is cleared, see qsd_id_blocking_ast(): <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lqe_write_lock(lqe); <span class="code-keyword">if</span> (lustre_handle_equal(&amp;lockh, &amp;lqe-&gt;lqe_lockh)) { <span class="code-comment">/* Clear lqe_lockh &amp; reset qunit to 0 */</span> qsd_set_qunit(lqe, 0); memset(&amp;lqe-&gt;lqe_lockh, 0, sizeof(lqe-&gt;lqe_lockh)); lqe-&gt;lqe_edquot = <span class="code-keyword">false</span>; rel = <span class="code-keyword">true</span>; } lqe_write_unlock(lqe); </pre> </div></div></li> </ul> <ul class="alternate" type="square"> <li>While just after the lqe_lockh is cleared, reply of pending DQACQ is received, the the lqe_lockh is set back to old lockh again, see qsd_req_completion(): <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* Set the lqe_lockh */</span> <span class="code-keyword">if</span> (lustre_handle_is_used(lockh) &amp;&amp; !lustre_handle_equal(lockh, &amp;lqe-&gt;lqe_lockh)) lustre_handle_copy(&amp;lqe-&gt;lqe_lockh, lockh); </pre> </div></div></li> </ul> <p>Then pre-acquire will always try to get lock with stale lockh, and fail with -ENLOCK.</p> <p>But I didn't see why it cause crash in lqe64_hash_keycmp(), anyway, I'll cooke a patch to fix the race first.</p> <p>patch to fix the -ENOLCK error: <a href="http://review.whamcloud.com/8322" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8322</a></p> <p>Looks my analysis about the ENOLCK isn't quite right: the DQACQ should have held lock reference, so lock can't be canceled when there is pending DQACQ. Mahmoud, do you know what kind of operations could cause the problem? Is your source tree same as our source tree?</p> <p>Niu</p> <p>NASA do include some patches. There is a link to their tree in the Environment field above</p> <p>Peter</p> <p>We had another crash. We do have a crash dump. But needs to be only view by US Citizen.</p> <p>I tried to get more info on the DQACQ error. Got a list of users that are generating the error. And then looked at there quota. Found some odd things. Some users are have used files count much less than what the are using. for example </p> <ol> <li>find /nobackupp7/ppritche/ -type f -user ppritche| wc -l<br/> 22986</li> </ol> <ol> <li>lfs quota -u ppritche /nobackupp7<br/> Disk quotas for user ppritche (uid 11563):<br/> Filesystem kbytes quota limit grace files quota limit grace<br/> /nobackupp7 2040023252 2300000000 2600000000 - 30 75000 150000 -</li> </ol> <p>lfs quota says he is using 30 files but find show &gt;22000 files. there are other examples like this.</p> <p>I am able to look at this Crashdump as I have done before. I send an email. </p> <p>Thank you, Keith, let's see if there is anything new in the new crash.</p> <blockquote> <p>Some users are have used files count much less than what the are using</p></blockquote> <p>Is this a new system or upgraded from old system? Will 'lfs quota' show the newly created files? Can quotacheck fix the inconsistence problem? ('tune2fs -O ^quota' to disable quota then 'tune2fs -O quota' to enable quota, with uptodate e2fsprogs, see <a href="https://jira.whamcloud.com/browse/LU-3861" title="Quota issues after upgrade from 2.1.4 to 2.4" class="issue-link" data-issue-key="LU-3861"><del>LU-3861</del></a>)</p> <p>This is a filesystem that was upgraded from 2.1.5 to 2.4. We initially ran into this file undercount and did the tune2fs command. One thing to note about these users. Their files are initially copied as root to this filesystem. They their ownership is changed. Here is an example</p> <p>/nobackupp7 &lt;-Top level directory<br/> /nobackupp7/ARRIVALS.anudelma &lt;-- Owned by user:anudelma<br/> /nobackupp7/ARRIVALS.anudelma/jliu7 &lt;-- owned by jlui7 and all files under this dir.</p> <ol> <li>lfs quota -u jliu7 /nobackupp7<br/> Disk quotas for user jliu7 (uid 11925):<br/> Filesystem kbytes quota limit grace files quota limit grace<br/> /nobackupp7 4340116920 0 0 - 0 0 0 -</li> </ol> <p>So when the owner ship is changed the file are not accounted for.</p> <p>Question Using tune2fs:<br/> Once we unmount the ost and run 'tune2fs -O ^quota', do we need to mount and umount it before running 'tune2fs -O quota'</p> <p>Keith,<br/> I uploaded the core file see you email for info</p> <p>Mahmoud</p> <p>I will take a good look this crash and report back. </p> <blockquote> <p>So when the owner ship is changed the file are not accounted for.</p></blockquote> <p>Is this reproduceable? Any error messages in the log?</p> <blockquote> <p>Question Using tune2fs:<br/> Once we unmount the ost and run 'tune2fs -O ^quota', do we need to mount and umount it before running 'tune2fs -O quota'</p></blockquote> <p>No, no need to do mount and umount.</p> <p>Keith,</p> <p>We need some update on this? Any luck extracting info from the core files?</p> <p>I have taken over this from Kieth, very sorry about the delay. I have the dump unpacked, and should be able to get you some information this week. </p> <p>I see you are using a non-standard build - would it be possible for you to unload your vmlinux for this crash dump? </p> <p>I upload the <a href="https://jira.whamcloud.com/browse/LU-4249" title="exception RIP: lqe64_hash_keycmp+12" class="issue-link" data-issue-key="LU-4249"><del>LU-4249</del></a>RPMS.tgz to ftp.whamcloud.com which will have the needed file you need.</p> <p>Cliff,<br/> Have you had a chance to look at the crashdump? We hit this bug 3 time this morning.</p> <p>Thanks,<br/> Mahmoud</p> <p>Mahmoud, Cliff has already checked the crashdump, and the crashdump shows that it crashed in same place as the first crash you posted in this ticket.</p> <p>I'm not sure if it's related to the cfs_hash bug, but you'd apply e44670a0301d3edf13ebd1bc728ad4c797369de2 (<a href="https://jira.whamcloud.com/browse/LU-4362" title="cfs_hash_rehash_key() passed wrong parameters to cfs_hash_keycpy" class="issue-link" data-issue-key="LU-4362"><del>LU-4362</del></a>)</p> <p>We will apply <a href="https://jira.whamcloud.com/browse/LU-4362" title="cfs_hash_rehash_key() passed wrong parameters to cfs_hash_keycpy" class="issue-link" data-issue-key="LU-4362"><del>LU-4362</del></a> and test.</p> <p>Thanks.</p> <p>Looks like <a href="https://jira.whamcloud.com/browse/LU-4362" title="cfs_hash_rehash_key() passed wrong parameters to cfs_hash_keycpy" class="issue-link" data-issue-key="LU-4362"><del>LU-4362</del></a> DID not fix this crash. We just had 3 oss crash at the same time. Looks like this is very sensitive to mds recovery.</p> <p>ustreError: 13476:0:(qsd_handler.c:344:qsd_req_completion()) Skipped 9 previous similar messages^M<br/> LustreError: 167-0: nbp7-MDT0000-lwp-OST0002: This client was evicted by nbp7-MDT0000; in progress operations using this service will fail.^M<br/> LustreError: Skipped 1 previous similar message^M<br/> Lustre: Evicted from MGS (at 10.151.27.38@o2ib) after server handle changed from 0xc262b1e69b1d10fc to 0xe69d9b7903f78200^M<br/> Lustre: MGC10.151.27.38@o2ib: Connection restored to MGS (at 10.151.27.38@o2ib)^M<br/> Lustre: Skipped 2 previous similar messages^M<br/> Lustre: 82396:0:(qsd_reint.c:365:qsd_reconciliation()) nbp7-OST000e: failed to report quota. <span class="error">&#91;0x200000006:0x20000:0x0&#93;</span>, -37^M<br/> Lustre: 82396:0:(qsd_reint.c:525:qsd_reint_main()) nbp7-OST000e: reconciliation failed. <span class="error">&#91;0x0:0x0:0x0&#93;</span>, -37^M<br/> Lustre: nbp7-OST0002: deleting orphan objects from 0x0:7609774 to 0x0:7610145^M<br/> Lustre: nbp7-OST000e: deleting orphan objects from 0x0:7567992 to 0x0:7568097^M<br/> Lustre: nbp7-OST001a: deleting orphan objects from 0x0:7599891 to 0x0:7600225^M<br/> general protection fault: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP ^M<br/> last sysfs file: /sys/devices/pci0000:40/0000:40:03.2/0000:44:00.1/host4/rport-4:0-0/target4:0:0/4:0:0:27/state^M<br/> 15 out of 16 cpus in kdb, waiting for the rest, timeout in 10 second(s)^M<br/> ...1 cpu is not in kdb, its state is unknown^M<br/> ^M<br/> Entering kdb (current=0xffff880bdf4a3540, pid 18116) on processor 0 Oops: (null)^M<br/> due to oops @ 0xffffffffa0c59f2c^M<br/> r15 = 0x5a5a5a5a5a5a5a5a r14 = 0x0000000000000000 ^M<br/> r13 = 0xffff880950324f60 r12 = 0xffff880be47436c0 ^M<br/> bp = 0xffff880be4743630 bx = 0xffff8806048505c0 ^M<br/> r11 = 0x0000000000000000 r10 = 0x00000000000007ca ^M<br/> r9 = 0x0000000000000001 r8 = 0x0000000000000003 ^M<br/> ax = 0x0000000000002ccb cx = 0x0000000000000000 ^M<br/> dx = 0x0000000000000000 si = 0x5a5a5a5a5a5a5a5a ^M<br/> di = 0xffff880950324f60 orig_ax = 0xffffffffffffffff ^M<br/> ip = 0xffffffffa0c59f2c cs = 0x0000000000000010 ^M<br/> flags = 0x0000000000010206 sp = 0xffff880be4743620 ^M<br/> ss = 0x0000000000000018 &amp;regs = 0xffff880be4743588^M</p> <p>from the assembly code of lqe64_hask_keycmp, the problem should be the lquota_entry managed by cfs_hash is destroyed,</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">0xffffffffa0bf8f2c &lt;lqe64_hash_keycmp+12&gt;: cmp %rax,0x10(%rsi) &lt;-- 0x10(%rsi) is the <span class="code-quote">"lqe-&gt;lqe_id.qid_uid"</span> </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [exception RIP: lqe64_hash_keycmp+12] RIP: ffffffffa0bf8f2c RSP: ffff88058e5736d0 RFLAGS: 00010206 RAX: 0000000000002f7d RBX: ffff880fd6450e00 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 5a5a5a5a5a5a5a5a RDI: ffff88058f1b2f60 RBP: ffff88058e5736d0 R8: 0000000000000003 R9: 0000000000000000 R10: ffff88058f1b2f28 R11: ffffffffa0cac2f0 R12: ffff88058e573760 R13: ffff88058f1b2f60 R14: 0000000000000000 R15: 5a5a5a5a5a5a5a5a </pre> </div></div> <p>the RSI is 0x5a5a5a5a5a5a5a5a, which means the corresponding lquota_entry is freed.</p> <p>Can we expect a patch soon. This is our most critical issue.</p> <p>Hi Mahmoud, </p> <p>One possible scenario is that the quota code screws up the refcount on the lquota_entry structure, causing a lqe to be freed while still referenced in the hash table. We are working on a diagnostic patch under <a href="http://review.whamcloud.com/#/c/9070" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9070</a>. We would like to make sure it passes our regression tests before you apply it. Stay tuned.</p> <p>We got a crash and here is the debug output.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 167-0: nbp7-MDT0000-lwp-OST004a: This client was evicted by nbp7-MDT0000; in progress operations using <span class="code-keyword">this</span> service will fail. Lustre: nbp7-MDT0000-lwp-OST003e: Connection restored to nbp7-MDT0000 (at 10.151.27.38@o2ib) Lustre: Evicted from MGS (at 10.151.27.38@o2ib) after server handle changed from 0x77507a47a4c36f84 to 0x5127d6849d7f5d1c Lustre: nbp7-MDT0000-lwp-OST004a: Connection restored to nbp7-MDT0000 (at 10.151.27.38@o2ib) Lustre: 45157:0:(qsd_reint.c:365:qsd_reconciliation()) nbp7-OST0032: failed to report quota. [0x200000006:0x20000:0x0], -37 Lustre: 45157:0:(qsd_reint.c:525:qsd_reint_main()) nbp7-OST0032: reconciliation failed. [0x0:0x0:0x0], -37 Lustre: 45187:0:(qsd_reint.c:365:qsd_reconciliation()) nbp7-OST0026: failed to report quota. [0x200000006:0x20000:0x0], -37 Lustre: 45187:0:(qsd_reint.c:525:qsd_reint_main()) nbp7-OST0026: reconciliation failed. [0x0:0x0:0x0], -37 Lustre: 45188:0:(qsd_reint.c:365:qsd_reconciliation()) nbp7-OST0042: failed to report quota. [0x200000006:0x20000:0x0], -37 Lustre: 45188:0:(qsd_reint.c:525:qsd_reint_main()) nbp7-OST0042: reconciliation failed. [0x0:0x0:0x0], -37 Lustre: 45189:0:(qsd_reint.c:365:qsd_reconciliation()) nbp7-OST002a: failed to report quota. [0x200000006:0x20000:0x0], -37 Lustre: 45189:0:(qsd_reint.c:525:qsd_reint_main()) nbp7-OST002a: reconciliation failed. [0x0:0x0:0x0], -37 Lustre: nbp7-OST0026: deleting orphan objects from 0x0:16242960 to 0x0:16242977 Lustre: nbp7-OST002a: deleting orphan objects from 0x0:16088539 to 0x0:16088557 Lustre: nbp7-OST002e: deleting orphan objects from 0x0:15914638 to 0x0:15914671 Lustre: nbp7-OST003a: deleting orphan objects from 0x0:15238739 to 0x0:15238764 Lustre: nbp7-OST0042: deleting orphan objects from 0x0:15604504 to 0x0:15604524 Lustre: nbp7-OST003e: deleting orphan objects from 0x0:16407770 to 0x0:16407841 Lustre: nbp7-OST004a: deleting orphan objects from 0x0:15822277 to 0x0:15822318 Lustre: nbp7-OST004e: deleting orphan objects from 0x0:15730335 to 0x0:15730375 Lustre: nbp7-OST0036: deleting orphan objects from 0x0:14913994 to 0x0:14914030 Lustre: nbp7-OST0046: deleting orphan objects from 0x0:15378324 to 0x0:15378342 Lustre: nbp7-OST0032: deleting orphan objects from 0x0:15902313 to 0x0:15902368 Lustre: nbp7-OST0052: deleting orphan objects from 0x0:15714908 to 0x0:15714926 Lustre: 45191:0:(qsd_reint.c:365:qsd_reconciliation()) nbp7-OST0046: failed to report quota. [0x200000006:0x20000:0x0], -37 Lustre: 45191:0:(qsd_reint.c:525:qsd_reint_main()) nbp7-OST0046: reconciliation failed. [0x0:0x0:0x0], -37 Lustre: 45209:0:(qsd_reint.c:365:qsd_reconciliation()) nbp7-OST003e: failed to report quota. [0x200000006:0x20000:0x0], -37 Lustre: 45209:0:(qsd_reint.c:525:qsd_reint_main()) nbp7-OST003e: reconciliation failed. [0x0:0x0:0x0], -37 LustreError: 11901:0:(lquota_internal.h:272:lqe_putref()) $$$ Freeing quota entry <span class="code-keyword">while</span> it is still referenced in the hash!!! qsd:nbp7-OST002a qtype:usr id:11920 enforced:1 granted:31459296 pending:0 waiting:0 req:0 usage:31396440 qunit:4194304 qtune:524288 edquot:0 LustreError: 11901:0:(lquota_internal.h:273:lqe_putref()) LBUG </pre> </div></div> <p>There should be a stack trace dumped after the LBUG. Could you please paste it here?</p> <p>This is from a second host that crashed at the same time. It has the call trace</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">ustre: nbp7-MDT0000-lwp-OST003b: Connection restored to nbp7-MDT0000 (at 10.151.27.38@o2ib) Lustre: Evicted from MGS (at 10.151.27.38@o2ib) after server handle changed from 0x77507a47a4c430e4 to 0x5127d6849d7f8850 Lustre: nbp7-MDT0000-lwp-OST0033: Connection restored to nbp7-MDT0000 (at 10.151.27.38@o2ib) Lustre: 47116:0:(qsd_reint.c:365:qsd_reconciliation()) nbp7-OST003f: failed to report quota. [0x200000006:0x20000:0x0], -37 Lustre: 47116:0:(qsd_reint.c:525:qsd_reint_main()) nbp7-OST003f: reconciliation failed. [0x0:0x0:0x0], -37 Lustre: 47117:0:(qsd_reint.c:365:qsd_reconciliation()) nbp7-OST002f: failed to report quota. [0x200000006:0x20000:0x0], -37 Lustre: 47117:0:(qsd_reint.c:525:qsd_reint_main()) nbp7-OST002f: reconciliation failed. [0x0:0x0:0x0], -37 Lustre: 47122:0:(qsd_reint.c:365:qsd_reconciliation()) nbp7-OST004f: failed to report quota. [0x200000006:0x20000:0x0], -37 Lustre: 47122:0:(qsd_reint.c:525:qsd_reint_main()) nbp7-OST004f: reconciliation failed. [0x0:0x0:0x0], -37 Lustre: nbp7-OST002b: deleting orphan objects from 0x0:16291625 to 0x0:16291649 Lustre: nbp7-OST003b: deleting orphan objects from 0x0:16000564 to 0x0:16000656 Lustre: nbp7-OST0027: deleting orphan objects from 0x0:16520477 to 0x0:16520523 Lustre: nbp7-OST0033: deleting orphan objects from 0x0:16359404 to 0x0:16359437 Lustre: nbp7-OST0047: deleting orphan objects from 0x0:16321198 to 0x0:16321227 Lustre: nbp7-OST0043: deleting orphan objects from 0x0:16268525 to 0x0:16268561 Lustre: nbp7-OST002f: deleting orphan objects from 0x0:15751870 to 0x0:15751911 Lustre: nbp7-OST003f: deleting orphan objects from 0x0:16136079 to 0x0:16136110 Lustre: nbp7-OST004b: deleting orphan objects from 0x0:15683379 to 0x0:15683400 Lustre: nbp7-OST004f: deleting orphan objects from 0x0:16583505 to 0x0:16583534 Lustre: nbp7-OST0037: deleting orphan objects from 0x0:16228767 to 0x0:16228810 Lustre: nbp7-OST0053: deleting orphan objects from 0x0:16334906 to 0x0:16334966 Lustre: 47127:0:(qsd_reint.c:365:qsd_reconciliation()) nbp7-OST004b: failed to report quota. [0x200000006:0x20000:0x0], -37 Lustre: 47127:0:(qsd_reint.c:525:qsd_reint_main()) nbp7-OST004b: reconciliation failed. [0x0:0x0:0x0], -37 LustreError: 12423:0:(lquota_internal.h:272:lqe_putref()) $$$ Freeing quota entry <span class="code-keyword">while</span> it is still referenced in the hash!!! qsd:nbp7-OST003f qtype:usr id:9637 enforced:1 granted:2111482660 pending:0 waiting:0 req:0 usage:2111481984 qunit:4096 qtune:1024 edquot:0 LustreError: 12423:0:(lquota_internal.h:273:lqe_putref()) LBUG Pid: 12423, comm: lquota_wb_nbp7- Call Trace: [&lt;ffffffffa0486895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa0486e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0c761ef&gt;] qsd_upd_thread+0xc8f/0xdd0 [lquota] [&lt;ffffffff81063be0&gt;] ? default_wake_function+0x0/0x20 Entering kdb (current=0xffff881023ab4040, pid 12423) on processor 24 Oops: (<span class="code-keyword">null</span>) due to oops @ 0x0 </pre> </div></div> <p>I didn't see how quota code screwed the refcount so far, will continue to investigate it further. Johann, do you have any idea? Thanks.</p> <p>One thing confused me is that: qsd_reconciliation() reported "failed to report quota for -ENOLCK", which means qsd_reconciliation() called qsd_adjust() and returned -ENOLCK, however I didn't see the corresponding error message from qsd_req_completion() in the log:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* despite -EDQUOT &amp; -EINPROGRESS errors, the master might still * grant us back quota space to adjust quota overrun */ <span class="code-keyword">if</span> (ret != 0 &amp;&amp; ret != -EDQUOT &amp;&amp; ret != -EINPROGRESS) { <span class="code-keyword">if</span> (ret != -ETIMEDOUT &amp;&amp; ret != -ENOTCONN &amp;&amp; ret != -ESHUTDOWN &amp;&amp; ret != -EAGAIN) <span class="code-comment">/* print errors only <span class="code-keyword">if</span> <span class="code-keyword">return</span> code is unexpected */</span> LQUOTA_ERROR(lqe, <span class="code-quote">"DQACQ failed with %d, flags:0x%x"</span>, ret, reqbody-&gt;qb_flags); GOTO(out, ret); } </pre> </div></div> <p>Hi, Mahmoud</p> <p>I didn't see why "DQACQ failed with -37" wasn't printed in the log, is it possible that the running code isn't same as 2.4.0-4nasS? Do you have other 2.4 systems are also suffering this problem? Thanks.</p> <p>As Johann pointed, the error message could probably be flooded. Could you enable quota debug (D_QUOTA) on all OSTs and provide lustre logs as well when it's reproduced next time? Thank you.</p> <p>We have a second filesystem running the same code (2.4.1-5.2nasS) It doesn't have any of the ENOLCK errors. The filesystem with the errors was a 2.1.5 that was upgraded to 2.4. The other started out with 2.4. We do want to track down the ENOLCK errors.</p> <p>We have D_QUOTA enabled. But when we see a crash the host panics and we are not able to get a debug dump.</p> <p>Additional info: These past few crashes have been after the MDS crashed and finished recovery. Then we would see 1 or more OSS crash. But we have see this crash without the MDS crash.</p> <p>I took a current debug dump of all OSS and MDS and uploaded to the ftp site. (nbp7.debug.tgz)</p> <p>Hi, we hit same issue (filed as <a href="https://jira.whamcloud.com/browse/LU-4807" title="repeating DQACQ failed with -37" class="issue-link" data-issue-key="LU-4807"><del>LU-4807</del></a>) of this problem. The following error messages have been showing up every 10 minute, even now. turning on quota debug on OSS and collecting lustre debug might help?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Mar 25 00:29:51 ddnoss4 kernel: LustreError: 4503:0:(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x2 qsd:home2-OST0026 qtype:grp id:3303 enforced:1 granted:7364460 pending:0 waiting:0 req:1 usage:7165364 qunit:4194304 qtune:524288 edquot:0 Mar 25 00:39:51 ddnoss4 kernel: LustreError: 4503:0:(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x2 qsd:home2-OST0026 qtype:grp id:3303 enforced:1 granted:7364460 pending:0 waiting:0 req:1 usage:7165364 qunit:4194304 qtune:524288 edquot:0 Mar 25 00:49:51 ddnoss4 kernel: LustreError: 4503:0:(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x2 qsd:home2-OST0026 qtype:grp id:3303 enforced:1 granted:7364460 pending:0 waiting:0 req:1 usage:7165364 qunit:4194304 qtune:524288 edquot:0 Mar 25 00:59:51 ddnoss4 kernel: LustreError: 4503:0:(qsd_handler.c:344:qsd_req_completion()) $$$ DQACQ failed with -37, flags:0x2 qsd:home2-OST0026 qtype:gr </pre> </div></div> <p>Hi, Ihara</p> <p>If the OSS is still showing such error message every 10 minutes, collecting debug logs (it's better to have both MDT and OST log with D_QUOTA enabled) can help us to figure out why dqacq failed with -ENOLCK. Thank you.</p> <p>Seems there is a small race window in qsd_intent_lock(), I updated the patch.</p> <p>Hi Niu, </p> <p>uploaded debug log on ftp.whamcloud.com/uploads/<a href="https://jira.whamcloud.com/browse/LU-4249" title="exception RIP: lqe64_hash_keycmp+12" class="issue-link" data-issue-key="LU-4249"><del>LU-4249</del></a><br/> Hope this help for analysis.</p> <p>Hi, Ihara/Mahmoud<br/> Could you apply the debug patch (<a href="http://review.whamcloud.com/#/c/9808/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9808/</a> ) to try to collect more information when the bug is hit again? (please make sure the autotest passed before apply the patch, and the previous debug patch needs be reverted before apply this one). Thank you.</p> <p>Niu, do you want me to undo patch at <a href="http://review.whamcloud.com/#/c/9070" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9070</a> or to keep it, in addition to #9808? Thanks!</p> <p>The #9070 needs to be reversed before applying #9808.</p> <p>However, there are still conflicts in three files. I believe your patch was not generated against 2.4 branch. The tree files with conflicts were:</p> <p>lustre/quota/qmt_pool.c<br/> lustre/quota/qsd_lock.c<br/> lustre/quota/qsd_writeback.c</p> <p>One conflict in qmt_pool.c at qmt_pool_lqe_lookup():</p> <p> if (qid-&gt;qid_uid == 0) </p> { /* caller wants to access grace time, no need to look up the * entry since we keep a reference on ID 0 all the time */ lqe = pool-&gt;qpi_grace_lqe[qtype]; &lt;&lt;&lt;&lt;&lt;&lt;&lt; HEAD lqe_getref(lqe); GOTO(out, 0); ======= lqe_getref(lqe, LQE_REF_IDX_MAX); GOTO(out, lqe); &gt;&gt;&gt;&gt;&gt;&gt;&gt; 6ec5b50... LU-4249 quota: lqe reference debug patch } <p> /* now that we have the pool, let's look-up the quota entry in the</p> <ul> <li>right quota site */</li> </ul> <p>The top of trunk of 2.4 branch shuold have GOTO line like this:<br/> GOTO(out, 0);<br/> but yours is<br/> GOTO(out,lqe);</p> <p>I think one example is enough for you to see the difference.</p> <p>Sorry, I made the patch against master mistakenly, here is the patch for b2_4: <a href="http://review.whamcloud.com/9833" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9833</a></p> <p>This new patch doesn't address the conflict in lustre/quota/qsd_writeback.c, because there is a quota fix introduced in 2.4.2 (<a href="https://jira.whamcloud.com/browse/LU-3460" title="recovery-small test_51 timeout: lqe_iter_cb(): Inuse quota entry" class="issue-link" data-issue-key="LU-3460"><del>LU-3460</del></a> <a href="http://review.whamcloud.com/#/c/8169/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8169/</a>), I suggest you apply this fix first.</p> <p>Hi Niu, I cherry-picked <a href="https://jira.whamcloud.com/browse/LU-3460" title="recovery-small test_51 timeout: lqe_iter_cb(): Inuse quota entry" class="issue-link" data-issue-key="LU-3460"><del>LU-3460</del></a> and then #9833. It applied clean now. Thanks!</p> <p>Hi Javed</p> <p>Looks your post isn't related to this bug, could you open another ticket to track it? Thanks.</p> <p>thanks niu. deleted from here.</p> <p>Hi Niu, so, finally, what exactly patches should we apply againt b2_4 if we want enabled debug patches?</p> <blockquote> <p>Hi Niu, so, finally, what exactly patches should we apply againt b2_4 if we want enabled debug patches?</p></blockquote> <p>fix of <a href="https://jira.whamcloud.com/browse/LU-3460" title="recovery-small test_51 timeout: lqe_iter_cb(): Inuse quota entry" class="issue-link" data-issue-key="LU-3460"><del>LU-3460</del></a> <a href="http://review.whamcloud.com/#/c/8169/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8169/</a> and debug patch <a href="http://review.whamcloud.com/9833" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9833</a></p> <p>Niu, we applied patches, and then, same error messages have been showing up. Attahced are recent debug log and syslog messages after applies patches.</p> <p>Hi, Ihara, the ENOLCK error message problem is addressed in another ticket, please see <a href="https://jira.whamcloud.com/browse/LU-4920" title="lqe_lockh should always be cleared on id lock cancel" class="issue-link" data-issue-key="LU-4920"><del>LU-4920</del></a>.</p> <p>This debug patch is to collect information on lqe refcount when system crash on "exception RIP: ...", please keep the debug patch applied until the crash problem reproduced. Thanks.</p> <p><a href="http://review.whamcloud.com/10988" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10988</a></p> <p>b2_4: <a href="http://review.whamcloud.com/11019" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11019</a><br/> b2_5: <a href="http://review.whamcloud.com/11020" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11020</a></p> <p>Landed for 2.7</p> <p>Do we need to undo #9833 (the debug patch) and apply #11019?<br/> Please advise.</p> <p>Also please land #11019 and #11020.</p> <p>I will leave it to Niu to comment definitively about whether the debug patch needs to be removed but my understanding is that #11019 is intended to be the fix for b2_4. The fixes for maintenance branches will be landed when a release is scheduled for those branches.</p> <p>But wouldn't that cause your engineers who work in future patches have different code base from ours with accumulated b2_4/b2_5 patches?</p> <p>Well, even without this happening it would still be quite possible that the release you baseline on lags behind the tip of the maintenance branch. If such a situation arises where there are multiple "floating" patches that affect the same area then we can always assist in resolving the conflict. I would think that introducing patch dependencies should make this easier though. It's probably simpler to talk about this on the next call than back and forth in a ticket <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <blockquote> <p>Do we need to undo #9833 (the debug patch) and apply #11019?<br/> Please advise.</p></blockquote> <p>Yes, that debug patch a little bit heavier than the former one (but it provides more detailed information), it wasn't supposed to be carried in production system all the time. Thanks.</p> Duplicate LU-4807 Related LU-4633 Development Rank 1|hzw8t3: Rank (Obsolete) 11583 Severity