https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6470 Lustre <p>Running SWL test on Hyperion, multiple clients timeout, eventually are evicted due to lock timeouts. <br/> Typical client stack:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">INFO: task ior:76875 blocked <span class="code-keyword">for</span> more than 120 seconds. Not tainted 2.6.32-431.29.2.el6.x86_64 #1 <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. ior D 0000000000000000 0 76875 76869 0x00000000 ffff88083f5ddd18 0000000000000082 0000000000000000 ffff8808341ab1d8 ffff88083f5ddc88 ffffffff81227e9f ffff88083f5ddd68 ffffffff81199045 ffff880871a9b058 ffff88083f5ddfd8 000000000000fbc8 ffff880871a9b058 Call Trace: [&lt;ffffffff81227e9f&gt;] ? security_inode_permission+0x1f/0x30 [&lt;ffffffff81199045&gt;] ? __link_path_walk+0x145/0x1000 [&lt;ffffffff8152a5be&gt;] __mutex_lock_slowpath+0x13e/0x180 [&lt;ffffffff8152a45b&gt;] mutex_lock+0x2b/0x50 [&lt;ffffffff8119ba76&gt;] do_filp_open+0x2d6/0xd20 [&lt;ffffffff811bd6b8&gt;] ? do_statfs_native+0x98/0xb0 [&lt;ffffffff8128f83a&gt;] ? strncpy_from_user+0x4a/0x90 [&lt;ffffffff811a8b82&gt;] ? alloc_fd+0x92/0x160 [&lt;ffffffff81185be9&gt;] do_sys_open+0x69/0x140 [&lt;ffffffff81185d00&gt;] sys_open+0x20/0x30 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Server side:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Apr 16 12:45:55 iws5 kernel: LustreError: 0:0:(ldlm_lockd.c:341:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 192.168.124.165@o2ib ns: filter-lustre-OST0024_UUID lock: ffff8801eac5e740/0x9c91b8d7046afd8 lrc: 3/0,0 mode: PR/PR res: [0x1d8c6:0x0:0x0].0 rrc: 13 type: EXT [0-&gt;18446744073709551615] (req 29796335616-&gt;29930553343) flags: 0x60000000010020 nid: 192.168.124.165@o2ib remote: 0x42e28ecfaef2b33a expref: 6 pid: 109819 timeout: 4391206162 lvb_type: 0 Apr 16 13:45:23 iws3 kernel: LustreError: 0:0:(ldlm_lockd.c:341:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.124.165@o2ib ns: filter-lustre-OST002d_UUID lock: ffff8805ccaf1180/0xc2e6f2e60c6a3a1f lrc: 3/0,0 mode: PR/PR res: [0x28183:0x0:0x0].0 rrc: 10 type: EXT [0-&gt;18446744073709551615] (req 29527900160-&gt;29662117887) flags: 0x60000000010020 nid: 192.168.124.165@o2ib remote: 0x42e28ecfaef2c102 expref: 5 pid: 23687 timeout: 4394807701 lvb_type: 0 </pre> </div></div> <p>Maybe related to DDN-56? Easy to reproduce if more data is required <br/> I dumped the lustre log from a client immediately after an eviction, file attached</p> Hyperion, 2.7.52 tag - ldiskfs format 200 clients LU-6470

SWL tests appear to wedge on mutex, clients are evicted

Bug Major Resolved Cannot Reproduce WC Triage Cliff White Thu, 16 Apr 2015 20:58:30 +0000 Sat, 9 Oct 2021 06:44:43 +0000 Sat, 9 Oct 2021 06:44:43 +0000 Lustre 2.8.0 0 3 <p>Dmesg from client after eviction. Client was evicted by multiple OSTs</p> <p>the client backtraces are indicating that MDS is stuck doing something.<br/> So it would be great to get MDS side of the story from MDS logs, would that still be possible?</p> <p>Cliff, is this running ZFS on the MDT/OST or ldiskfs? It wouldn't be surprising if there is a heavy metadata load on a ZFS MDT, but more surprising if it is ldiskfs. Could you please fill in the FSTYPE and client count into the Environment. Is there also racer/tar/dbench running on other clients during IOR?</p> <p>Also, getting the stack traces from the client would be useful, since even if a client thread is blocked waiting for the MDS (which is true from all the stack traces shown) it shouldn't prevent lock callbacks from the OSTs from being processed.</p> <p>The setup has already been torn down, but if I get a repeat i will get client traces. The failure was on ldiskfs, currently testing with ZFS</p> <p>Ah, the stack trace and lustre log dump from the client are already attached - the client is iwc115, see the two attachments. </p> <p>I have re-checked the logs, and the only errors in that time period were from the OSS nodes, no MDS errors.</p> Development Rank 1|hzxaxb: Rank (Obsolete) 9223372036854775807 Severity