https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7016 Lustre <p>When it reads multiple files from HSM archive using posix copytool, it causes deadlock on the client.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Aug 18 07:46:58 r38 kernel: INFO: task lhsmtool_posix:16414 blocked for more than 120 seconds. Aug 18 07:46:58 r38 kernel: Not tainted 2.6.32-431.29.2.el6.x86_64 #1 Aug 18 07:46:58 r38 kernel: "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 18 07:46:58 r38 kernel: lhsmtool_posi D 0000000000000002 0 16414 14111 0x00000000 Aug 18 07:46:58 r38 kernel: ffff880f6379bd18 0000000000000082 ffff88102609502a ffff881020055c40 Aug 18 07:46:58 r38 kernel: ffff880f6379bc88 ffffffff81227e9f ffff880f6379bd68 ffffffff81199045 Aug 18 07:46:59 r38 kernel: ffff880f67361af8 ffff880f6379bfd8 000000000000fbc8 ffff880f67361af8 Aug 18 07:46:59 r38 kernel: Call Trace: Aug 18 07:46:59 r38 kernel: [&lt;ffffffff81227e9f&gt;] ? security_inode_permission+0x1f/0x30 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff81199045&gt;] ? __link_path_walk+0x145/0x1000 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8152a5be&gt;] __mutex_lock_slowpath+0x13e/0x180 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8152a45b&gt;] mutex_lock+0x2b/0x50 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8119ba76&gt;] do_filp_open+0x2d6/0xd20 Aug 18 07:46:59 r38 kernel: [&lt;ffffffffa0605343&gt;] ? ll_file_release+0x683/0xac0 [lustre] Aug 18 07:46:59 r38 kernel: [&lt;ffffffff81153ea1&gt;] ? unlink_anon_vmas+0x71/0xd0 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8128f83a&gt;] ? strncpy_from_user+0x4a/0x90 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff811a8b82&gt;] ? alloc_fd+0x92/0x160 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff81185be9&gt;] do_sys_open+0x69/0x140 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff81185d00&gt;] sys_open+0x20/0x30 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b Aug 18 07:46:59 r38 kernel: INFO: task lhsmtool_posix:16415 blocked for more than 120 seconds. Aug 18 07:46:59 r38 kernel: Not tainted 2.6.32-431.29.2.el6.x86_64 #1 Aug 18 07:46:59 r38 kernel: "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 18 07:46:59 r38 kernel: lhsmtool_posi D 0000000000000004 0 16415 14111 0x00000000 Aug 18 07:46:59 r38 kernel: ffff880f634c1d18 0000000000000082 0000000000000000 ffff881020055c40 Aug 18 07:46:59 r38 kernel: ffff880f634c1c88 ffffffff81227e9f ffff880f634c1d68 ffffffff81199045 Aug 18 07:46:59 r38 kernel: ffff880f64e5a5f8 ffff880f634c1fd8 000000000000fbc8 ffff880f64e5a5f8 Aug 18 07:46:59 r38 kernel: Call Trace: Aug 18 07:46:59 r38 kernel: [&lt;ffffffff81227e9f&gt;] ? security_inode_permission+0x1f/0x30 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff81199045&gt;] ? __link_path_walk+0x145/0x1000 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8152a5be&gt;] __mutex_lock_slowpath+0x13e/0x180 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8152a45b&gt;] mutex_lock+0x2b/0x50 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8119ba76&gt;] do_filp_open+0x2d6/0xd20 Aug 18 07:46:59 r38 kernel: [&lt;ffffffffa0605343&gt;] ? ll_file_release+0x683/0xac0 [lustre] Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8128f83a&gt;] ? strncpy_from_user+0x4a/0x90 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff811a8b82&gt;] ? alloc_fd+0x92/0x160 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff81185be9&gt;] do_sys_open+0x69/0x140 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff81185d00&gt;] sys_open+0x20/0x30 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b Aug 18 07:46:59 r38 kernel: INFO: task md5sum:16312 blocked for more than 120 seconds. Aug 18 07:46:59 r38 kernel: Not tainted 2.6.32-431.29.2.el6.x86_64 #1 Aug 18 07:46:59 r38 kernel: "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 18 07:46:59 r38 kernel: md5sum D 0000000000000001 0 16312 15803 0x00000000 Aug 18 07:46:59 r38 kernel: ffff881fbd611b58 0000000000000082 0000000000000000 0000000000000286 Aug 18 07:46:59 r38 kernel: ffff881fbd611b68 ffff881020055c40 ffff881fbd611b38 ffffffffa05eb3f0 Aug 18 07:46:59 r38 kernel: ffff882026a85ab8 ffff881fbd611fd8 000000000000fbc8 ffff882026a85ab8 Aug 18 07:46:59 r38 kernel: Call Trace: Aug 18 07:46:59 r38 kernel: [&lt;ffffffffa05eb3f0&gt;] ? ll_lookup_finish_locks+0x270/0x8d0 [lustre] Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8152a5be&gt;] __mutex_lock_slowpath+0x13e/0x180 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8152a45b&gt;] mutex_lock+0x2b/0x50 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff811989ab&gt;] do_lookup+0x11b/0x230 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff811996a4&gt;] __link_path_walk+0x7a4/0x1000 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8114a3d7&gt;] ? handle_pte_fault+0xf7/0xb00 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8104eeb7&gt;] ? pte_alloc_one+0x37/0x50 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8119a1ba&gt;] path_walk+0x6a/0xe0 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8119a3cb&gt;] filename_lookup+0x6b/0xc0 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff81226d56&gt;] ? security_file_alloc+0x16/0x20 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8119b8a4&gt;] do_filp_open+0x104/0xd20 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8114e820&gt;] ? __vma_link_rb+0x30/0x40 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8114e8cb&gt;] ? vma_link+0x9b/0xf0 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8128f83a&gt;] ? strncpy_from_user+0x4a/0x90 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff811a8b82&gt;] ? alloc_fd+0x92/0x160 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff81185be9&gt;] do_sys_open+0x69/0x140 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff81185d00&gt;] sys_open+0x20/0x30 Aug 18 07:46:59 r38 kernel: [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> master LU-7016

deadlock on reading multiple files from HSM archive

Bug Critical Resolved Cannot Reproduce Bruno Faccini Shuichi Ihara Mon, 17 Aug 2015 22:50:40 +0000 Wed, 13 Oct 2021 03:06:12 +0000 Wed, 13 Oct 2021 03:06:12 +0000 Lustre 2.8.0 0 5 <p>this is reproduder.<br/> the problem always happens when reading (md5sum) files </p> <p>client's syslog</p> <p>Hello Ihara,<br/> According to the syslog you have provided, and the hung threads stacks dumped in it, both your reproducer/script, and the md5sum commands, are running on the same node than the Posix CopyTool. So, the first question that comes to my mind is, does your copytool use the same Lustre mount-point than the others Lustre Client-side commands ??<br/> And if yes, can you give a new try to your reproducer with the Posix CopyTool using a separate mount-point ?</p> <p>Yes, test script and copytool are running on same node. is that problem?<br/> let me quick run on separate nodes. Thanks for your advise.</p> <p>Thanks in advance, but you can also use a separate+specific Lustre mount-point for the copytool.</p> <p>OK, confirmed. At least, it works if we use separate nodes for copytool and test script.</p> <p>Any more work required on this ticket or can we mark it as resolved?</p> <p>Thanks,<br/> ~ jfc.</p> <p>John, even if the dual/2nd specific mount-point work-around works, I still need to work with the reproducer provided and understand/fix the dead-lock condition.</p> Development End date Mon, 31 Aug 2015 22:50:40 +0000 Rank 1|hzxknz: Rank (Obsolete) 9223372036854775807 Severity Start date Mon, 17 Aug 2015 22:50:40 +0000