https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4365 Lustre <p>recovery-small test_51 hung as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Failing mds1 on client-30vm3 CMD: client-30vm3 grep -c /mnt/mds1' ' /proc/mounts Stopping /mnt/mds1 (opts:) on client-30vm3 CMD: client-30vm3 umount -d /mnt/mds1 </pre> </div></div> <p>LBUG occurred on MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>23:04:50:Lustre: DEBUG MARKER: umount -d /mnt/mds1 23:04:50:LustreError: 7276:0:(ldlm_resource.c:804:ldlm_resource_complain()) lustre-MDT0000-lwp-MDT0000: namespace resource [0x200000006:0x1010000:0x0].0 (ffff8800709d3980) refcount nonzero (1) after lock cleanup; forcing cleanup. 23:04:50:LustreError: 7276:0:(ldlm_resource.c:804:ldlm_resource_complain()) Skipped 1 previous similar message 23:04:50:LustreError: 7276:0:(ldlm_resource.c:1415:ldlm_resource_dump()) --- Resource: [0x200000006:0x1010000:0x0].0 (ffff8800709d3980) refcount = 2 23:04:50:LustreError: 7276:0:(ldlm_resource.c:1418:ldlm_resource_dump()) Granted locks (in reverse order): 23:04:50:LustreError: 7276:0:(ldlm_resource.c:1421:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-lwp-MDT0000 lock: ffff880063768b40/0x3b627f62d662915 lrc: 2/1,0 mode: CR/CR res: [0x200000006:0x1010000:0x0].0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x3b627f62d66293f expref: -99 pid: 6945 timeout: 0 lvb_type: 2 23:04:50:LustreError: 7276:0:(ldlm_resource.c:1421:ldlm_resource_dump()) Skipped 1 previous similar message 23:04:50:LustreError: 7276:0:(ldlm_resource.c:1415:ldlm_resource_dump()) --- Resource: [0x200000006:0x10000:0x0].0 (ffff8800709d3ac0) refcount = 2 23:04:50:LustreError: 7276:0:(ldlm_resource.c:1418:ldlm_resource_dump()) Granted locks (in reverse order): 23:04:50:LustreError: 7276:0:(ldlm_lib.c:2137:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery 23:04:50:Lustre: 6947:0:(ldlm_lib.c:1801:target_recovery_overseer()) recovery is aborted, evict exports in recovery 23:04:50:Lustre: 6947:0:(ldlm_lib.c:1801:target_recovery_overseer()) Skipped 2 previous similar messages 23:04:50:LustreError: 6937:0:(osp_precreate.c:737:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 23:04:50:LustreError: 7276:0:(qsd_lib.c:294:qsd_qtype_fini()) ASSERTION( atomic_read(&amp;qqi-&gt;qqi_ref) == 1 ) failed: 23:04:50:LustreError: 7276:0:(qsd_lib.c:294:qsd_qtype_fini()) LBUG 23:04:50:Pid: 7276, comm: umount 23:04:50: 23:04:50:Call Trace: 23:04:50: [&lt;ffffffffa05c2895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 23:04:50: [&lt;ffffffffa05c2e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 23:04:51: [&lt;ffffffffa0efcc23&gt;] qsd_qtype_fini+0x363/0x3a0 [lquota] 23:04:51: [&lt;ffffffffa0efcfc8&gt;] qsd_fini+0xd8/0x460 [lquota] 23:04:51: [&lt;ffffffffa0d751a8&gt;] osd_shutdown+0x38/0xe0 [osd_zfs] 23:04:51: [&lt;ffffffffa0d78571&gt;] osd_process_config+0x141/0x190 [osd_zfs] 23:04:51: [&lt;ffffffffa0843b41&gt;] lod_process_config+0x4e1/0x17a0 [lod] 23:04:51: [&lt;ffffffffa05d32d1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 23:04:51: [&lt;ffffffff8100bb8e&gt;] ? apic_timer_interrupt+0xe/0x20 23:04:52: [&lt;ffffffffa118ed00&gt;] mdd_process_config+0x210/0x610 [mdd] 23:04:52: [&lt;ffffffffa1722c56&gt;] mdt_stack_fini+0x176/0xbe0 [mdt] 23:04:52: [&lt;ffffffffa1189950&gt;] ? mdd_init_capa_ctxt+0x120/0x130 [mdd] 23:04:52: [&lt;ffffffffa1723dca&gt;] mdt_device_fini+0x70a/0xd90 [mdt] 23:04:52: [&lt;ffffffffa071fbc7&gt;] class_cleanup+0x577/0xda0 [obdclass] 23:04:52: [&lt;ffffffffa06f4b06&gt;] ? class_name2dev+0x56/0xe0 [obdclass] 23:04:52: [&lt;ffffffffa07214ac&gt;] class_process_config+0x10bc/0x1c80 [obdclass] 23:04:52: [&lt;ffffffffa05cdd98&gt;] ? libcfs_log_return+0x28/0x40 [libcfs] 23:04:52: [&lt;ffffffffa071ad11&gt;] ? lustre_cfg_new+0x391/0x7e0 [obdclass] 23:04:52: [&lt;ffffffffa07221e9&gt;] class_manual_cleanup+0x179/0x6f0 [obdclass] 23:04:52: [&lt;ffffffffa06f4b06&gt;] ? class_name2dev+0x56/0xe0 [obdclass] 23:04:52: [&lt;ffffffffa07571dc&gt;] server_put_super+0x5bc/0xf00 [obdclass] 23:04:52: [&lt;ffffffff8118366b&gt;] generic_shutdown_super+0x5b/0xe0 23:04:52: [&lt;ffffffff81183756&gt;] kill_anon_super+0x16/0x60 23:04:52: [&lt;ffffffffa07240a6&gt;] lustre_kill_super+0x36/0x60 [obdclass] 23:04:52: [&lt;ffffffff81183ef7&gt;] deactivate_super+0x57/0x80 23:04:52: [&lt;ffffffff811a21ef&gt;] mntput_no_expire+0xbf/0x110 23:04:52: [&lt;ffffffff811a2c5b&gt;] sys_umount+0x7b/0x3a0 23:04:52: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b 23:04:52: 23:04:52:Kernel panic - not syncing: LBUG </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/beee4bb4-60cb-11e3-bd66-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/beee4bb4-60cb-11e3-bd66-52540035b04c</a></p> <br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/63/">http://build.whamcloud.com/job/lustre-b2_4/63/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs<br/> LU-4365

recovery-small test_51: LBUG: (qsd_lib.c:294:qsd_qtype_fini()) ASSERTION( atomic_read(&qqi->qqi_ref) == 1 ) failed

Bug Blocker Resolved Fixed Niu Yawei Jian Yu Mon, 9 Dec 2013 15:36:57 +0000 Tue, 31 Dec 2013 15:38:02 +0000 Tue, 17 Dec 2013 16:43:17 +0000 Lustre 2.4.1 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.4.2 Lustre 2.5.1 0 7 <p>The same test <b>passed</b> in another ZFS full group test session on Lustre b2_4 build #63:<br/> <a href="https://maloo.whamcloud.com/test_sets/40487e62-5fb6-11e3-85c5-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/40487e62-5fb6-11e3-85c5-52540035b04c</a></p> <p>By searching on Maloo, I found all of the previous recovery-small test 51 failures were <a href="https://jira.whamcloud.com/browse/LU-3460" title="recovery-small test_51 timeout: lqe_iter_cb(): Inuse quota entry" class="issue-link" data-issue-key="LU-3460"><del>LU-3460</del></a>. Is this a potential regression introduced by the fix of <a href="https://jira.whamcloud.com/browse/LU-3460" title="recovery-small test_51 timeout: lqe_iter_cb(): Inuse quota entry" class="issue-link" data-issue-key="LU-3460"><del>LU-3460</del></a>?</p> <p>Niu</p> <p>Could you please comment?</p> <p>Thanks</p> <p>Peter</p> <p>Hi Niu, </p> <p>I have a question about qsd_glb_blocking_ast(), in which</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">case</span> LDLM_CB_CANCELING: { struct qsd_qtype_info *qqi; LDLM_DEBUG(lock, <span class="code-quote">"canceling global quota lock"</span>); qqi = qsd_glb_ast_data_get(lock, <span class="code-keyword">true</span>); <span class="code-keyword">if</span> (qqi == NULL) <span class="code-keyword">break</span>; ... lu_ref_del(&amp;qqi-&gt;qqi_reference, <span class="code-quote">"ast_data_get"</span>, lock); qqi_putref(qqi); </pre> </div></div> <p>since it calls qsd_glb_ast_data_get(lock, true) to reset the lock-&gt;l_ast_data and in it the qqi's ref has been put, why here blocking ast put its ref again?</p> <p>While in qsd_glb_glimpse_ast(), it calls qsd_glb_ast_data_get(lock, false) and qqi's ref is kept.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/67/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/67/</a><br/> Distro/Arch: RHEL6.4/x86_64 (server), SLES11SP2/x86_64 (client)</p> <p>sanity-scrub test 3 also hit this LBUG:<br/> <a href="https://maloo.whamcloud.com/test_sets/4bdfe456-6347-11e3-8c76-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4bdfe456-6347-11e3-8c76-52540035b04c</a></p> <p>Hi, Bobi</p> <p>The blocking ast calls qsd_glb_ast_data_get(lock, true) to put the qqi ref which holding by the lock (since the lock is goning to be canceled), and then put the ref which just got in qsd_glb_ast_data_get() (for caller). In qsd_glb_glimpse_ast(), it only needs to put the ref holding for caller.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">23:04:50:LustreError: 7276:0:(ldlm_resource.c:804:ldlm_resource_complain()) lustre-MDT0000-lwp-MDT0000: namespace resource [0x200000006:0x1010000:0x0].0 (ffff8800709d3980) refcount nonzero (1) after lock cleanup; forcing cleanup. 23:04:50:LustreError: 7276:0:(ldlm_resource.c:804:ldlm_resource_complain()) Skipped 1 previous similar message 23:04:50:LustreError: 7276:0:(ldlm_resource.c:1415:ldlm_resource_dump()) --- Resource: [0x200000006:0x1010000:0x0].0 (ffff8800709d3980) refcount = 2 23:04:50:LustreError: 7276:0:(ldlm_resource.c:1418:ldlm_resource_dump()) Granted locks (in reverse order): 23:04:50:LustreError: 7276:0:(ldlm_resource.c:1421:ldlm_resource_dump()) ### ### ns: lustre-MDT0000-lwp-MDT0000 lock: ffff880063768b40/0x3b627f62d662915 lrc: 2/1,0 mode: CR/CR res: [0x200000006:0x1010000:0x0].0 rrc: 2 type: PLN flags: 0x1106400000000 nid: local remote: 0x3b627f62d66293f expref: -99 pid: 6945 timeout: 0 lvb_type: 2 23:04:50:LustreError: 7276:0:(ldlm_resource.c:1421:ldlm_resource_dump()) Skipped 1 previous similar message 23:04:50:LustreError: 7276:0:(ldlm_resource.c:1415:ldlm_resource_dump()) --- Resource: [0x200000006:0x10000:0x0].0 (ffff8800709d3ac0) refcount = 2 23:04:50:LustreError: 7276:0:(ldlm_resource.c:1418:ldlm_resource_dump()) Granted locks (in reverse order): </pre> </div></div> <p>The qqi reference is holding by the global quota locks, and the two quota global locks were not cleanup because the lock reader was not 0, but I don't see who is holding the lock.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">23:04:38:LustreError: 5385:0:(qsd_reint.c:58:qsd_reint_completion()) lustre-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x1010000:0x0], rc:-5 23:04:38:LustreError: 5385:0:(qsd_reint.c:58:qsd_reint_completion()) Skipped 1 previous similar message </pre> </div></div> <p>I see there were some lock enqueue failures in the log, maybe there is some bug in the error path of lock enqueue which caused the lock not dropped properly?</p> <p>Looks there is a race: On server umount, if quota reint process is just holding the global locks for reintegraion, the locks will not be cleared by server_put_super() <del>&gt; lustre_disconnect_lwp(), and later on when qsd_fini() stops reint process, the global locks will be dropped, however, because lock cancel is done asynchronously, when the qsd_fini() reachs LASSERT(cfs_atomic_read(&amp;qqi</del>&gt;qqi_ref) == 1), the global locks haven't been canceled yet and they still holding qqi refcount.</p> <p>seems we'd wait a while for such case in qsd_fini(), I'll try to cooke a patch soon.</p> <p>patch for master: <a href="http://review.whamcloud.com/8583" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8583</a></p> <p>patch for b2_4: <a href="http://review.whamcloud.com/8586" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8586</a></p> <p>Patch landed on master branch for 2.6.0 and b2_4 branch for 2.4.2.</p> <p>Patches have landed. Can this ticket be closed, or is more work expected/needed in this ticket?</p> <p>Landed for 2.4.2 and 2.5. Will land to 2.5.1 when work starts on that release</p> Development Rank 1|hzwaxz: Rank (Obsolete) 11948 Severity