https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12916 Lustre <p>Allow small/unaligned writes to be placed directly into pagecache on the OST without starting IO. This should help IOPS for small writes where we are not limited by pagecache performance as is the case for large writes (where patch <a href="https://review.whamcloud.com/34422" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34422</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-12071" title="bypass pagecache for large files" class="issue-link" data-issue-key="LU-12071"><del>LU-12071</del></a> osd-ldiskfs: bypass pagecache if requested</tt>" is trying to do the opposite and <b>avoid</b> page cache for large writes).</p> <p>A reasonable starting point would be 64KiB writes, since this is the upper limit for short writes (data embedded into the BRW RPC request), and beyond this we get good performance for larger writes already (per <a href="https://www.eofs.eu/_media/events/lad19/05_andreas_dilger-lustre_2.14_and_beyond.pdf" class="external-link" target="_blank" rel="nofollow noopener">testing of flash at different io sizes</a>, page 8).</p> LU-12916

allow small/unaligned writes into writeback cache if not O_SYNC

Improvement Minor Open Unresolved Alex Zhuravlev Andreas Dilger Tue, 29 Oct 2019 23:03:20 +0000 Mon, 22 Jan 2024 15:49:53 +0000 0 8 <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a patch: <a href="https://review.whamcloud.com/36596" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36596</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12916" title="allow small/unaligned writes into writeback cache if not O_SYNC" class="issue-link" data-issue-key="LU-12916">LU-12916</a> osd: use writeback cache in ldiskfs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 18<br/> Commit: eb24c28a61953800c5dd9f382620d573e99f1f43</p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38747" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38747</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12916" title="allow small/unaligned writes into writeback cache if not O_SYNC" class="issue-link" data-issue-key="LU-12916">LU-12916</a> osd: send data to journal<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f767bc5689bbdd53ae6ef03e660c7ac18bb4e5a7</p> <p>I thought the original driver for turning off the page cache was maximum IOPS.  I can't tag him (Only WC users can tag people in WC Jira, for some reason...), but I think Ihara did the benchmarking at the time.  Can someone tag him here?</p> <p>Patrick, there are two different goals/needs here. There is a need to write directly into page cache for poorly formed workloads like IO500 ior-hard-write, because the writes are not page aligned, and doing synchronous read-modify-write for sub-page IO size is very expensive, even with a block-level write cache in front of NVMe. </p> <p>We've already instituted a dynamic cache at the OST level, where large aligned writes are never cached, and page cache is disabled for non-rotational devices, but we need to re-enable page cache for those badly-formed writes. </p> <p>Hi Alex,<br/> I am studying your patch (<a href="https://review.whamcloud.com/36596" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36596</a>) recently. It seems that your patch used delayed allocation with write-back in order to obtain better performance.</p> <p>According to the description in URL: <a href="https://lwn.net/Articles/322823/" class="external-link" target="_blank" rel="nofollow noopener">https://lwn.net/Articles/322823/</a></p> <blockquote><p>"Delayed allocation" means that the filesystem tries to delay the allocation of physical disk blocks for written data for as long as possible. This policy brings some important performance benefits. Many files are short-lived; delayed allocation can keep the system from writing fleeting temporary files to disk at all. And, for longer-lived files, delayed allocation allows the kernel to accumulate more data and to allocate the blocks for data contiguously, speeding up both the write and any subsequent reads of that data. It's an important optimization which is found in most contemporary filesystems.</p></blockquote> <blockquote><p>But, if blocks have not been allocated for a file, there is no need to write them quickly as a security measure. Since the blocks do not yet exist, it is not possible to read somebody else's data from them. So ext4 will not (cannot) write out unallocated blocks as part of the next journal commit cycle. Those blocks will, instead, wait until the kernel decides to flush them out; at that point, physical blocks will be allocated on disk and the data will be made persistent. The kernel doesn't like to let file data sit unwritten for too long, but it can still take a minute or so (with the default settings) for that data to be flushed - far longer than the five seconds normally seen with ext3. And that is why a crash can cause the loss of quite a bit more data when ext4 is being used.</p></blockquote> <p>Delayed allocation is a good feature: "delayed allocation allows the kernel to accumulate more data and to allocate the blocks for data contiguously, speeding up both the write"<br/> But I am afraid we can not use delayed allocation in write-back support for our OSD-ldiskfs...<br/> "Ext4 will not (cannot) write out unallocated blocks as part of the next journal commit cycle..."<br/> Thus during the journal commit, the page with unallocated blocks is not committed into the stable disk...<br/> If the last_committed &gt; transno of the bulk write on the client, we may wrongly release the unstable page of the write RPC on the client which may be not stable on the storage on the server...</p> <p>So I think we still need to allocate blocks for each write page on OST, after that we can write each page in write-back way with ordered-journal mode.<br/> I have no idea how it will help with the performance.<br/> But I expect it can improve the performance of ior-hard-write (combined with Partick's unaligned DIO patch) and mdtest-hard-write(combined with my patch <a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/49342/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/49342/</a>).</p> <p>I could help to refine your patch if you are busy. However, I am not very familiar with Ext4, you are the expert in ldiskfs, could you please give your professional opinion?</p> <p>Thanks,<br/> Qian</p> <p>"Qian Yingjin &lt;qian@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50687" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50687</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12916" title="allow small/unaligned writes into writeback cache if not O_SYNC" class="issue-link" data-issue-key="LU-12916">LU-12916</a> osd: use writeback for small writes in ldiskfs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b6c208f1bd2c220e926f555ad78d4331b911231c</p> <p>"Qian Yingjin &lt;qian@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50940" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50940</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12916" title="allow small/unaligned writes into writeback cache if not O_SYNC" class="issue-link" data-issue-key="LU-12916">LU-12916</a> osd-ldiskfs: check and submit good full write<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a4f26530367f040ba148acecd198a6ea13541701</p> <p>"Qian Yingjin &lt;qian@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51033" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51033</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12916" title="allow small/unaligned writes into writeback cache if not O_SYNC" class="issue-link" data-issue-key="LU-12916">LU-12916</a> osd-ldiskfs: add delayed allocation support<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a3eae66329bc90bf6b00a1ba45b69ac63c1157f7</p> <p>"Qian Yingjin &lt;qian@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51063" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51063</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12916" title="allow small/unaligned writes into writeback cache if not O_SYNC" class="issue-link" data-issue-key="LU-12916">LU-12916</a> osd-ldiskfs: detect good extent via extent tree<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8365ad9fec435b712d61469102b6f52261898a1e</p> <p>"Qian Yingjin &lt;qian@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51159" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51159</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12916" title="allow small/unaligned writes into writeback cache if not O_SYNC" class="issue-link" data-issue-key="LU-12916">LU-12916</a> osd-ldiskfs: use workqueue to write good extent<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5f1f2e1dfbd5a18233302fdc7254649c461f5dfc</p> <p>master</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[RESULT] ior-hard-write 5.368100 GiB/s : time 313.086 seconds </pre> </div></div> <p>master + <a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/51159/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/51159/</a> (patch set6)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[RESULT] ior-hard-write 8.027850 GiB/s : time 319.704 seconds </pre> </div></div> <p>the patch demonstrated 50% performance improvements today. After patch, we are seeing very good aligned 1M IOs to disks, but it causes memory pressures on OSSs. When it happens, no more  aligned IOs, and the performance down in the end.<br/> I will give more detail stats to compare memory pressure vs performance, but patch still needs to release and reclaim after flsuh dirty data to disks. it should be able to keep nice aligned IOs to disks as much as possible.</p> <p>i wrote in email:</p> <blockquote> <p>I was thinking that it would be possible to drop the pages from page cache right after a large IO was submitted. That would leave more RAM available to aggregate small writes in cache until they can form a complete write, and avoid memory pressure. </p> <p>We might be able to do this at the osd-ldiskfs level, but it might need to patch the write completion callback to drop the pages from cache? That should be conditional on some flag on the inode or something so that we can control whether the writes are cached or not (eg. by size or tunable parameter).</p></blockquote> <p>Yingjin replied:</p> <blockquote> <p>We have already implemented this for full stripe write in the patch (<a href="https://review.whamcloud.com/51159/):" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/51159/):</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> osd_da_full_write_workfn() <span class="code-keyword">if</span> (osd-&gt;od_range_delalloc_drop_cache &amp;&amp; rc == 0 &amp;&amp;        nr_pages == mpd.locked_index + 1) {             pgoff_t end = gstart + nr_pages - 1;             <span class="code-object">int</span> err;             rc = filemap_fdatawait_range(mapping, gstart &lt;&lt; PAGE_SHIFT,                                end &lt;&lt; PAGE_SHIFT);             <span class="code-keyword">if</span> (rc &lt; 0) {                   CERROR(<span class="code-quote">"Wait writeback range failed: rc = %d\n"</span>, rc);                   GOTO(out_iput, rc);             }             err = invalidate_inode_pages2_range(mapping, gstart, end);             <span class="code-keyword">if</span> (err &lt; 0)                   CERROR(<span class="code-quote">"failed to invalidate pages: rc = %d\n"</span>, err);       } </pre> </div></div> <p>We use Ext4 extent_status tree to track the delayed allocation extents for a file.<br/> When a write in the I/O service thread with delayed allocation can form a good full stripe extent I/O, which means the extent_status extent after merge contains a full I/O extent where the offset and size in this full I/O extent are all 1MiB aligned, then we will launch a extra work queue thread to do dirty page flush in this I/O range.<br/> In this work queue thread, it will first flush this full stripe I/O. And if the osd-<tt>ldiskfs.&#42;.range_delalloc_drop_cache=1</tt>, then it will wait the full stripe I/O to finish and discard the pages from cache.</p> <p>However, under memory pressure with writeback enabled on OSD, the kernel will trigger page reclaim. It will call ext4|ldiskfs_writepages() to write out the dirty page, which may destroy the full extent I/O, resulting in lots of small I/O to the RAID disk system.<br/> Thus I think we may still need to patch the ldiskfs in ext4/ldiskfs_writepages(), add a flag to the inode with delalloc to indicate that it needs to wait for writeback finished and drop the cache to mitigate the memroy pressure, thus avoid the small I/O casued by kernel page reclaim mechanism as much as possible, reduce its impact on the performance.</p></blockquote> Related LU-13786 Development Rank 1|i00opz: Rank (Obsolete) 9223372036854775807