https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8736 Lustre <p>In latest soak-test, one of MDT stuck during umount</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 0-0: Forced cleanup waiting for soaked-MDT0000-osp-MDT0002 namespace with 1 resources in use, (rc=-110) </pre> </div></div> <p>The stack trace</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>umount S 0000000000000011 0 8015 8013 0x00000080 ffff8803d9b33808 0000000000000086 ffff8803d9b337d0 ffff8803d9b337cc ffff8803d9b33868 ffff88043fe84000 00001b24f314dc54 ffff880038635a00 00000000000003ff 0000000101c3089b ffff8803f3c31ad8 ffff8803d9b33fd8 Call Trace: [&lt;ffffffff8153a9b2&gt;] schedule_timeout+0x192/0x2e0 [&lt;ffffffff81089fa0&gt;] ? process_timeout+0x0/0x10 [&lt;ffffffffa0abded0&gt;] __ldlm_namespace_free+0x1c0/0x560 [ptlrpc] [&lt;ffffffff81067650&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0abe2df&gt;] ldlm_namespace_free_prior+0x6f/0x220 [ptlrpc] [&lt;ffffffffa13b0db2&gt;] osp_process_config+0x4a2/0x680 [osp] [&lt;ffffffff81291947&gt;] ? find_first_bit+0x47/0x80 [&lt;ffffffffa12c5650&gt;] lod_sub_process_config+0x100/0x1f0 [lod] [&lt;ffffffffa12cad66&gt;] lod_process_config+0x646/0x1580 [lod] [&lt;ffffffffa113e4ff&gt;] ? lfsck_stop+0x15f/0x4c0 [lfsck] [&lt;ffffffffa0801032&gt;] ? cfs_hash_bd_from_key+0x42/0xd0 [libcfs] [&lt;ffffffffa1343253&gt;] mdd_process_config+0x113/0x5e0 [mdd] [&lt;ffffffffa11fee62&gt;] mdt_device_fini+0x482/0x13e0 [mdt] [&lt;ffffffffa08df626&gt;] ? class_disconnect_exports+0x116/0x2f0 [obdclass] [&lt;ffffffffa08f82c2&gt;] class_cleanup+0x582/0xd30 [obdclass] [&lt;ffffffffa08dae56&gt;] ? class_name2dev+0x56/0xe0 [obdclass] [&lt;ffffffffa08fa5d6&gt;] class_process_config+0x1b66/0x24c0 [obdclass] [&lt;ffffffffa07fc151&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffff8117904c&gt;] ? __kmalloc+0x21c/0x230 [&lt;ffffffffa08fb3ef&gt;] class_manual_cleanup+0x4bf/0xc90 [obdclass] [&lt;ffffffffa08dae56&gt;] ? class_name2dev+0x56/0xe0 [obdclass] [&lt;ffffffffa092983c&gt;] server_put_super+0x8bc/0xcd0 [obdclass] [&lt;ffffffff81194aeb&gt;] generic_shutdown_super+0x5b/0xe0 [&lt;ffffffff81194bd6&gt;] kill_anon_super+0x16/0x60 [&lt;ffffffffa08fe596&gt;] lustre_kill_super+0x36/0x60 [obdclass] [&lt;ffffffff81195377&gt;] deactivate_super+0x57/0x80 [&lt;ffffffff811b533f&gt;] mntput_no_expire+0xbf/0x110 [&lt;ffffffff811b5e8b&gt;] sys_umount+0x7b/0x3a0 [&lt;ffffffff8100b0d2&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>And it seems there is a MDT handler thread (mdt_rename), which holds the remote lock on soaked-MDT0000-osp-MDT0002, but then stuck on local lock enqueue, which then block the namespace cleanup of umount.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mdt01_016 S 000000000000000a 0 7405 2 0x00000080 ffff8804027ab900 0000000000000046 0000000000000000 ffffffff810a1c1c ffff880433fef520 ffff8804027ab880 00000a768c137fd5 0000000000000000 ffff8804027ab8c0 0000000100ab043e ffff880433fefad8 ffff8804027abfd8 Call Trace: [&lt;ffffffff810a1c1c&gt;] ? remove_wait_queue+0x3c/0x50 [&lt;ffffffffa0ad54b0&gt;] ? ldlm_expired_completion_wait+0x0/0x250 [ptlrpc] [&lt;ffffffffa0ada07d&gt;] ldlm_completion_ast+0x68d/0x9b0 [ptlrpc] [&lt;ffffffff81067650&gt;] ? default_wake_function+0x0/0x20 [&lt;ffffffffa0ad93fe&gt;] ldlm_cli_enqueue_local+0x21e/0x810 [ptlrpc] [&lt;ffffffffa0ad99f0&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] [&lt;ffffffffa11fa770&gt;] ? mdt_blocking_ast+0x0/0x2e0 [mdt] [&lt;ffffffffa12074a4&gt;] mdt_object_local_lock+0x3a4/0xb00 [mdt] [&lt;ffffffffa11fa770&gt;] ? mdt_blocking_ast+0x0/0x2e0 [mdt] [&lt;ffffffffa0ad99f0&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] [&lt;ffffffffa1208103&gt;] mdt_object_lock_internal+0x63/0x320 [mdt] [&lt;ffffffffa1218e9e&gt;] ? mdt_lookup_version_check+0x9e/0x350 [mdt] [&lt;ffffffffa1208580&gt;] mdt_reint_object_lock+0x20/0x60 [mdt] [&lt;ffffffffa121cba7&gt;] mdt_reint_rename_or_migrate+0x1317/0x2690 [mdt] [&lt;ffffffffa11fa770&gt;] ? mdt_blocking_ast+0x0/0x2e0 [mdt] [&lt;ffffffffa0ad99f0&gt;] ? ldlm_completion_ast+0x0/0x9b0 [ptlrpc] [&lt;ffffffffa09238c0&gt;] ? lu_ucred+0x20/0x30 [obdclass] [&lt;ffffffffa0b06b00&gt;] ? lustre_pack_reply_v2+0xf0/0x280 [ptlrpc] [&lt;ffffffffa121df53&gt;] mdt_reint_rename+0x13/0x20 [mdt] [&lt;ffffffffa121704d&gt;] mdt_reint_rec+0x5d/0x200 [mdt] [&lt;ffffffffa1201d5b&gt;] mdt_reint_internal+0x62b/0xa50 [mdt] [&lt;ffffffffa120262b&gt;] mdt_reint+0x6b/0x120 [mdt] [&lt;ffffffffa0b6b0cc&gt;] tgt_request_handle+0x8ec/0x1440 [ptlrpc] [&lt;ffffffffa0b17821&gt;] ptlrpc_main+0xd31/0x1800 [ptlrpc] [&lt;ffffffff81539b0e&gt;] ? thread_return+0x4e/0x7d0 [&lt;ffffffffa0b16af0&gt;] ? ptlrpc_main+0x0/0x1800 [ptlrpc] [&lt;ffffffff810a138e&gt;] kthread+0x9e/0xc0 [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 [&lt;ffffffff810a12f0&gt;] ? kthread+0x0/0xc0 [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> LU-8736

stuck during umount in soak-test

Bug Minor In Progress Unresolved Lai Siyao Di Wang soak Wed, 19 Oct 2016 18:01:27 +0000 Mon, 18 Dec 2017 18:26:47 +0000 Lustre 2.9.0 0 6 <p>Cliff, can you file the above comment as a separate ticket?</p> <p>Hi Lai,</p> <p>Can you please look at the first issue in this ticket?</p> <p>Thanks.<br/> Joe</p> <p>Cliff, Di, <br/> For the initial problem, was MDT0002 already stopped or in the middle of unmounting at the time that MDT0000 got stuck? If it was up, were there any errors on the MDT0000 or MDT0002 console that indicate RPC timeouts or other errors? Could you please attach the MDS console logs from this time period. </p> <p>The other possibility is if this is a circular locking deadlock, which would need stack traces on both MDS nodes to see if there is another thread also stuck waiting for a lock. </p> <p>For the initial problem, no - MDT0002 is mounted on lola-10, and was not involved in the failover at all. It was neither stopped nor umounted</p> <p>console logs covering the time period</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>The other possibility is if this is a circular locking deadlock, which would need stack traces on both MDS nodes to see if there is another thread also stuck waiting for a lock. </pre> </div></div> <p>Yes, it looks like a circular locking deadlock here, see that mdt_reint_rename trace (on MDT2) I posted above, after the rename process on MDT2 got the remote lock(lock_A), then being blocked in local lock(lock_B) enqueue, which is probably holding by MDT0, then in the mean time umount happened, so MDT0 can not return the lock to MDT2 successfully, because of umount on MDT2 ( Can it? that is my guess, but need some investigation). Then we saw umount is hanging there for namespace cleanup because lock_A is still hold by the rename process.</p> <p>So it looks like we need re-order the umount process to make sure all of remote locks has been released before namespace cleanup. Need think a bit.</p> <p>Di, any further ideas on this?</p> Development Rank 1|hzysjz: Rank (Obsolete) 9223372036854775807 Severity