https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2078 Lustre <p>While running some single shared file IOR tests I noticed, with 'zpool iostat', that there were a lots write going on during the read phase. This was absolutely destroying performance and it wasn't at all clear to me why there should be any reads at all. IOR is just reading during this phase, no writes.</p> <p>On a hunch I set 'sync_on_lock_cancel=never' and the writes stopped entirely and performance improved at least 6x. I haven't looked at the code yet but I think there might be a few issues here.</p> <p>1) Why were so many lock revocations occurring during the read phase. I would have thought that each client would end up with a single concurrent read lock rather quickly. However, this behavior persistent for the 30 minutes it took to read all the data back.</p> <p>2) When revoking a read lock, or even a write lock which isn't actually covering any dirty data, there's no reason to issue the sync. I suspect this was never noticed because under ldiskfs if there's no dirty data to sync performing the sync is basically free. Under ZFS however if you call sync you will force the current txg to be written out. That will result in at a minimum the vdev labels at the beginning and end of each disk to be written. Lots of syncs will result each disk head basically seeking from the being to end of the disk repeatedly. Very very bad.</p> LU-2078

sync_on_lock_cancel too aggressive

Bug Minor Resolved Fixed Alex Zhuravlev Brian Behlendorf server topsequoia Thu, 19 Apr 2012 14:09:03 +0000 Thu, 26 Dec 2013 17:21:00 +0000 Mon, 8 Oct 2012 01:21:25 +0000 Lustre 2.4.0 0 4 <p>please, give this trivial patch a run?</p> <p><a href="http://review.whamcloud.com/#change,3355" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3355</a></p> <p>Per comments in Skype, it might make sense to remove the sync from the DLM level entirely, and implement a "commit on share" style handling of write data. That means the writes should not be committed to disk when the lock is cancelled, but rather only if the data is being read again. This should allow the initial writes will go faster because a larger number of writes can be aggregated into a single commit. If the data is not immediately being read back from the object, it may not even need a transaction commit.</p> <p>There is no data consistency problem from doing this, because until the data is read by another client it is in a Schrödinger's box, and whether it was committed to disk or not is irrelevant. Using the same "commit on share" mechanism also brings the OST data handling more in line with COS on the MDS.</p> <p>The most simplistic mechanism for this is to move the "inode version &gt; last_committed" conditional sync from change 3355 from the DLM cancel into the OST read code. This will improve the common case of many page-aligned writers on the same file that do not need to do read-modify-write of the file data. A more sophisticated mechanism would be to track the dirty extents of the file (via ZIL?), and only sync the inode and data if reading dirty pages.</p> <p>Brian, do you remember any details of the test? how many stripes? arguments to IOR, etc? thanks in advance.</p> <p>Alex, I was able to run a few single shared file tests today. Here's a summary of my results.</p> <p>I ran 4 separate tests. Two tests with our 55chaos tag (<b>without</b> <a href="http://review.whamcloud.com/3355" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3355</a>), one with sync_on_lock_cancel=always and another with sync_on_lock_cancel=never. Also, two tests were run with our 56chaos tag (<b>with</b> <a href="http://review.whamcloud.com/3355" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3355</a>), again one with "always" and another with "never".</p> <p>Also, each test was run using 64 client nodes, 1 task per node, striped over a single OST, using these IOR options "-a POSIX -CegkY".</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Single Shared File Read Performance (MiB/s) | always | never | --+---------+---------+ 55chaos | 469.19 | 1304.03 | --+---------+---------+ 56chaos | 988.26 | 1275.59 | --+---------+---------+ </pre> </div></div> <p>So just from the above data, your patch seems to provide a 2x improvement for reads!</p> <p>Also, I'll attach a tarball with the IOR output and and logs from MDS and OSS with '+dlmtrace' enabled during each test's read phase.</p> <p>IOR output for the read and write phase of each test. And MDS and OSS server logs for the read phase of each test with '+dlmtrace' enabled.</p> <p>thanks for the data Prakash. I'm still looking at dlmtrace.. it's still not clear why we don't reach<br/> the peak performance: the very first PR lock should commit the whole object and all subsequent reads<br/> should skip osd_sync().. so here is a debug patch: <a href="http://review.whamcloud.com/#change,3456" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3456</a><br/> please try, just current orion + this patch with sync_on_cancel=always should be enough. a log with<br/> +dlmtrace +rpctrace would be very useful. thanks in advance.</p> <p>This has the IOR output from the read and write phase of the test. Along with Lustre Logs from the OSS with +dlmtrace and +rpctrace.</p> <p>thanks.. here is another patch: <a href="http://review.whamcloud.com/#change,3456" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3456</a> <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> hope this time we'll get some improvement. again, a log with +dlmtrace +rpctrace is very welcome. thanks in advance.</p> <p>Same logs as before, except with our 58chaos tag (reverted debug patch-set 1 and applied debug patch-set 2 as requested).</p> <p>With this tag, I saw the best performance numbers of all my tests run so far (roughly equal to the sync_on_lock_cancel=never tests):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Single Shared File Read Performance (MiB/s) | always | never | --+---------+----------+ 58chaos | 1311.51 | untested | --+---------+----------+ </pre> </div></div> <p>So my question is, can we safely drop the write lock as is done in the debug patch? Or is this just a proof of concept?</p> <p>Also, What exactly is that lock protecting, and why?</p> <p>I went ahead and ran the same test with `sync_on_lock_cancel=never` as well. Here's the logs and performance numbers.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Single Shared File Read Performance (MiB/s) | always | never | --+---------+---------+ 58chaos | 1311.51 | 1265.46 | --+---------+---------+ </pre> </div></div> <p>Patch landed on master: <a href="http://review.whamcloud.com/4117" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4117</a></p> <p>Patch landed.</p> Related LU-2085 Development Epic server Project Rank 1|hzuwwf: Rank (Obsolete) 2992 Severity