https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15941 Lustre <p>This issue was created by maloo for Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/097393a8-5380-4d65-af83-5d44c963ce88" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/097393a8-5380-4d65-af83-5d44c963ce88</a></p> <p>test_398b failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Timeout occurred after 326 minutes, last suite running was sanity </pre> </div></div> <p>this started on June 10, after recent landing wave.<br/> I suspect two patches:</p> <ul> <li>c45b8a92a3 2022-05-11 | <a href="https://jira.whamcloud.com/browse/LU-15583" title="Update ZFS version to 2.1.2" class="issue-link" data-issue-key="LU-15583"><del>LU-15583</del></a> build: Update ZFS version to 2.1.2 <span class="error">&#91;Jian Yu&#93;</span></li> <li>b4880f3758 2021-07-15 | <a href="https://jira.whamcloud.com/browse/LU-15483" title="Minor DIO test improvements" class="issue-link" data-issue-key="LU-15483">LU-15483</a> tests: Improve test 398b <span class="error">&#91;Patrick Farrell&#93;</span></li> </ul> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_398b - Timeout occurred after 326 minutes, last suite running was sanity</p> LU-15941

sanity test_398b: timeouts with ZFS

Bug Minor Open Unresolved WC Triage Maloo Tue, 14 Jun 2022 06:17:41 +0000 Wed, 16 Aug 2023 12:03:38 +0000 0 5 <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/34d33d79-d068-4eeb-8990-9c8d06669a01" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/34d33d79-d068-4eeb-8990-9c8d06669a01</a></p> <p>It is reporting pathetic IOPS of 1, under 8 KB/s. I guess that is contention on the singe HDD on the host, also caused by r-m-w of the larger ZFS blocks? ALEX, do you think your blocksize patch <a href="https://review.whamcloud.com/47768" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47768</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-15963" title="sanityn test_56b: OSS OOM with ZFS" class="issue-link" data-issue-key="LU-15963">LU-15963</a> osd: use contiguous chunk to grow blocksize</tt>" might help this?</p> <p>I profiled 398b: dt_trans_stop() in ofd_commitrw_write() takes 50 usec with ldiskfs and 512831 with ZFS on average.<br/> the majority of OST_WRITE were missing OBD_BRW_ASYNC, this is why dt_trans_stop() was taking that long.<br/> changing max blocksize doesn't improve the situation significantly, locally at least, but I'm going to try that with AT.</p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/b7ffdf4c-d214-427b-95e0-379d8c837267" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b7ffdf4c-d214-427b-95e0-379d8c837267</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/cff8066f-91ae-4805-a4e2-ce35545e5bfe" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/cff8066f-91ae-4805-a4e2-ce35545e5bfe</a></p> <p>Alex,</p> <p>They are missing 'ASYNC' because they should be missing async - This is direct IO, which expects the server to do a sync each time.  This means DIO performance on ZFS is absolutely terrible.  And I think we can't fix it except by fixing our sync behavior on ZFS, which I understand is a huge project<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/help_16.png" height="16" width="16" align="absmiddle" alt="" border="0"/>.  So... yeah.</p> <p>IIRC, there are two significant performance issues with ZFS sync writes:</p> <ul class="alternate" type="square"> <li>one of course is the fact that transaction commit has a lot of overhead (4x überblock sync writes per device, full merkle tree flush each time)</li> <li>the other is that calling "sync" on ZFS does not actually <b>trigger</b> a transaction commit, it just waits for one to happen by itself. That is why, on average, Alex is reporting a 0.5s commit time, since the commit interval is 1s and half of the time we are close to the transaction already committing.</li> </ul> <p>For flash devices it would still be possible to commit thousands of times per second, and for HDD devices maybe 10/s, instead of 1/s. This would of course increase load on the storage and CPUs, but what else are they for, and why should both the clients and servers be waiting idle for the 1s ZFS transaction commit?</p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/2dda2437-8b99-4e36-b99d-f769947b2f6b" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/2dda2437-8b99-4e36-b99d-f769947b2f6b</a></p> Related LU-15963 Development Rank 1|i02s27: Rank (Obsolete) 9223372036854775807 Severity