https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16316 Lustre <p>We have experienced locks over the past few weeks on OSS based on ZFS 2.0.7, which makes the node unresponsive in terms of Lustre (OSS node goes <tt>unhealthy</tt>) and causes a huge load (&gt;400) on OSS. In some situations, directly after that, the load on MDS also increases, but it seems like a consequence of lost communication between MDS and affected OSS. We cannot associate this problem with the exact IO pattern or type of operation. We first address this problem here, but we cannot exclude that it should be addressed to ZFS developers - if you consider it, please let us know.  We attach two types of logs: the first from the 16th of October when both MDS and OSS were affected and the second from the 13th of November when only OSS was stuck. If you need more information, please don't hesitate to let us know.</p> <p> </p> <p>Regards</p> <p> </p> <p>Dominika Wanat </p> Lustre: 2.15.0_RC3, zfs 2.0.7 (both self-compiled) <br/> OS: Centos 8.5, kernel 4.18.0-348.7.1.el8_5.x86_64 LU-16316

ZFS OSS locks

Bug Major Open Unresolved WC Triage Dominika Wanat Wed, 16 Nov 2022 13:17:31 +0000 Fri, 23 Dec 2022 11:41:56 +0000 0 3 <blockquote><p>In some situations, directly after that, the load on MDS also increases, but it seems like a consequence of lost communication between MDS and affected OSS.</p></blockquote> <p>correct, this is because MDS gets stuck awaiting for new objects from the problem OST.</p> <p>I'm not 100% positive, but I found number of OST threads trying to prefetch data. you can try to disable prefetching to see whether it's related:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">echo 1 &gt; /sys/module/zfs/parameters/zfs_prefetch_disable</pre> </div></div> <p> &#8211; on OSTs</p> <p>Thanks for the hint. We are investigating the nodes with prefetch disabled. </p> <p>It looks like it helps - nodes have not hung since then. Do you consider fixing this behaviour of Lustre with ZFS prefetch enabled?</p> <blockquote><p>It looks like it helps - nodes have not hung since then. Do you consider fixing this behaviour of Lustre with ZFS prefetch enabled?</p></blockquote> <p>this is very workload specific thing.. we've seen number of reports where prefetching <em>does</em> improve performance.</p> Development Rank 1|i035vj: Rank (Obsolete) 9223372036854775807 Severity