https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15223 Lustre <p>Currently, when the user issues a partial page read/write, Lustre (including client side and server side) will convert it into a full page read/write.</p> <p>This is not efficient for small read/write operations , say if you want to read/write several bytes from a file, actually Lustre will  read/write a full page. This will become even worse for large PAGE_SIZE 64KB.</p> <p>Make Lustre do a real partial page read/write which the read/write range is actual from start to end(which are given by the user), so that we can get an efficient small read/write.</p> Arch: aarch64 (client) LU-15223

Improve partial page read/write

Improvement Minor In Progress Unresolved Xinliang Liu Xinliang Liu arm arm-server ppc Mon, 15 Nov 2021 04:03:26 +0000 Wed, 14 Jun 2023 15:40:20 +0000 Lustre 2.12.0 Lustre 2.12.4 0 6 <p>Copy partial page write comments from <a href="https://jira.whamcloud.com/browse/LU-11667" title="sanity test 317: FAIL: Expected Block 8 got 48 for f317.sanity" class="issue-link" data-issue-key="LU-11667"><del>LU-11667</del></a>.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=xinliang" class="user-hover" rel="xinliang">xinliang</a> added a comment  <del>16/Sep/21 11:17 AM</del>  edited<br/> Created two same size 10B  file in home dir and /mnt/lustre dir, if the backend filesystem block size is 4K. Then the inode allocated blocks should be the same( that is 8 if count by block size 512B).</p> <p>Test file created at home dir:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ getconf PAGESIZE 65536 $ echo <span class="code-quote">"123456789"</span> &gt; ~/testfile $ stat ~/testfile File: /root/testfile Size: 10 Blocks: 8 IO Block: 65536 regular file Device: fc02h/64514d Inode: 12863429 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2021-09-16 02:51:21.268641287 +0000 Modify: 2021-09-16 03:08:05.382557951 +0000 Change: 2021-09-16 03:08:05.382557951 +0000 Birth: - $ stat -c %b ~/testfile 8 $ stat -c %B ~/testfile 512 $ stat -c %s ~/testfile 10 $ stat -f ~/testfile File: <span class="code-quote">"/root/testfile"</span> ID: fc0200000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 52272379 Free: 45840170 Available: 45840170 Inodes: Total: 104549824 Free: 104176363 </pre> </div></div> <p>Test file created at Lustre dir:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ getconf PAGESIZE 65536 $ echo <span class="code-quote">"123456789"</span> &gt; /mnt/lustre/testfile $ stat -c %s /mnt/lustre/testfile 10 $ stat -c %B /mnt/lustre/testfile 512 $ stat -c %b /mnt/lustre/testfile 128 $ stat /mnt/lustre/testfile File: /mnt/lustre/testfile Size: 10 Blocks: 128 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272502274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2021-09-16 02:53:57.000000000 +0000 Modify: 2021-09-16 03:07:28.000000000 +0000 Change: 2021-09-16 03:07:28.000000000 +0000 Birth: - $ stat -f /mnt/lustre/testfile File: <span class="code-quote">"/mnt/lustre/testfile"</span> ID: 2c54f96600000000 Namelen: 255 Type: lustre Block size: 4096 Fundamental block size: 4096 Blocks: Total: 78276 Free: 77931 Available: 71141 Inodes: Total: 100000 Free: 99726 </pre> </div></div> <p>But the Lustre test file's inode blocks is 128. This should be wrong?</p> <p> <br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a> added a comment - 16/Sep/21 2:55 PM<br/> I think this has always been the case for writes from 64KB PAGE_SIZE clients (e.g. back to ia64). The reason is that the client sends a full-page write, because it is only tracking dirty pages, and the server writes the full amount of data sent by the client. I suspect that ext4 is handling this by having multiple 4KB buffer_heads on a 64KB page, and using the buffer dirty state to determine which pages to write, but Lustre doesn't use buffer heads.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=xinliang" class="user-hover" rel="xinliang">xinliang</a> added a comment - 29/Oct/21 5:55 PM<br/> HI Andreas,</p> <p>I found that this issue happens at Arm 64K PAGE_SIZE OST server.</p> <p>When create a file, blocks are allocated with PAGE_SIZE aligned, see function osd_ldiskfs_map_inode_pages().</p> <p>E.g. for 64K PAGE_SIZE Arm64 OST server, if create a file with size less than 64K, it actually allocates 128 blocks each block 512 Bytes.</p> <p>We need to adjust the test for 64K PAGE_SIZE OST server.<br/>  <br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=xinliang" class="user-hover" rel="xinliang">xinliang</a> added a comment - 04/Nov/21 9:09 AM<br/> I am thinking if we should make blocks allocation aligned with BLOCK_SIZE as ext4, which could save space for large PAGE_SIZE e.g. 64K. Then no need to make change to the test case. And I have a look at the code it seems both OSC client  and OST server need to adjust for this. The client always sends no hole pages (currently page start offset is always 0) to the server for writing now. And the server side needs to adjust making blocks allocation aligned with block size. <br/>  <br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=xinliang" class="user-hover" rel="xinliang">xinliang</a> added a comment - 04/Nov/21 6:02 PM<br/> Find out the client side page clip related code:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 298 <span class="code-object">int</span> osc_io_commit_async(<span class="code-keyword">const</span> struct lu_env *env, 299 <span class="code-keyword">const</span> struct cl_io_slice *ios, 300 struct cl_page_list *qin, <span class="code-object">int</span> from, <span class="code-object">int</span> to, 301 cl_commit_cbt cb) 302 { ... 315 <span class="code-comment">/* Handle partial page cases */</span> 316 last_page = cl_page_list_last(qin); 317 <span class="code-keyword">if</span> (oio-&gt;oi_lockless) { 318 page = cl_page_list_first(qin); 319 <span class="code-keyword">if</span> (page == last_page) { 320 cl_page_clip(env, page, from, to); 321 } <span class="code-keyword">else</span> { 322 <span class="code-keyword">if</span> (from != 0) 323 cl_page_clip(env, page, from, PAGE_SIZE); 324 <span class="code-keyword">if</span> (to != PAGE_SIZE) 325 cl_page_clip(env, last_page, 0, to); 326 } 327 } 328 329 ll_pagevec_init(pvec, 0); </pre> </div></div> <p>Currently, it seems a normal write don't go into this "if (oio-&gt;oi_lockless) {" part code. Anyone know why it is oi_lockless? @<a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="external-link" rel="nofollow">Andreas Dilger</a></p> <p> <br/>  <br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a> added a comment  <del>05/Nov/21 3:29 AM</del>  edited<br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=paf0186" class="user-hover" rel="paf0186">paf0186</a> you are probably the most interested in changing this code. Handling sub-page writes for ARM would not be very different than sub-page writes for x86, which would potentially allow eg. IO500 unaligned writes to be handled much more efficiently.<br/>  <br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=paf0186" class="user-hover" rel="paf0186">paf0186</a> added a comment - 05/Nov/21 3:36 AM<br/> Xinliang,</p> <p>I am not 100% sure I understand your question - Are you saying it <b>is</b> oi_lockless?  It should not be.  This (commit_async) code is buffered, and lockless buffered is broken and also off by default.  I have a patch to remove it, but it's normally off anyway.</p> <p>What are you looking for/hoping for here?</p> <p>Note we clip pages in other places too.<br/>  <br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=paf0186" class="user-hover" rel="paf0186">paf0186</a> added a comment - 05/Nov/21 3:47 AM<br/> "I am thinking if we should make blocks allocation aligned with BLOCK_SIZE as ext4, which could save space for large PAGE_SIZE e.g. 64K. Then no need to make change to the test case. And I have a look at the code it seems both OSC client  and OST server need to adjust for this. The client always sends no hole pages (currently page start offset is always 0) to the server for writing now. And the server side needs to adjust making blocks allocation aligned with block size. <br/>  "</p> <p>Can you talk more about what you're thinking?  I am not quite what the implication of changing block allocation on the server would be on the client.  Why does changing server block allocation filter back to the client like this?</p> <p>More generally, about partial page i/o:<br/> Generally speaking, we can't have partial pages except at the start and end of each write - that's a limitation of infiniband, but there are also page cache restrictions.</p> <p>In general, RDMA can be unaligned at the start, and unaligned at the end, but that's it.  This applies even when combining multiple RDMA regions - it's some limitation of the hardware/drivers.  So we have a truly unaligned I/O(with a partial page at beginning and end), but then we can't combine it with other I/Os.</p> <p>There is also a page cache limitation here.  The Linux page cache insists on working with full pages - It will only allow partial pages at file_size.  So, eg, a 3K file is a single page with 3K in it, and we can write just 3K.  But if we want to write 3K in to a large 'hole' in a file, Linux will enforce writing PAGE_SIZE.  This is not a restriction we can easily remove, it is an important part of the page cache.<br/>  <br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=paf0186" class="user-hover" rel="paf0186">paf0186</a> added a comment - 05/Nov/21 3:48 AM<br/> By the way, I am happy to keep talking about this, if you have thoughts or questions or whatever.  I've looked at sub-page I/O a few times, but you may have a different idea than what I have tried.<br/>  <br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a> added a comment - 05/Nov/21 5:01 AM<br/> Patrick, I was thinking that if we can handle a write (uncached) from the client that is RDMA 64KB, but has a non-zero start and end offset (4KB initially), it might be generalizable to any byte offset.</p> <p>I'm aware of the RDMA limitations, but I'm wondering if those can be bypassed (if necessary) by transferring a whole page over the network, but store it into a temporary page and copy the data for a cached/unaligned read-modify-write on the server to properly align the data. The content of the start/end of the page sent from the client would be irrelevant, since it will be trimmed by the server anyway when the copy is done</p> <p>While the copy might be expensive for very large writes, my expectation is that this would be most useful for small writes. That does raise the question of whether the data could be transferred in the RPC as a short write, but for GPU direct we require RDMA to send the data directly from the GPU RAM to the OSS. Maybe it is just a matter of generalizing the short write handling to allow copying from the middle of an RDMA page?<br/>  <br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=xinliang" class="user-hover" rel="xinliang">xinliang</a> added a comment  <del>06/Nov/21 11:10 PM</del>  edited<br/> Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=paf0186" class="user-hover" rel="paf0186">paf0186</a> and <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="external-link" rel="nofollow">Andreas Dilger</a>, thank you for the clarification about partial page write. It really helps me a lot.</p> <p>For ldiskfs backend filesystem,  I see that if the user issue a partial page cached write the Lustre (including client side and server side) will convert it in to a full page write. I want to make Lustre do a real partial page write inside which with the length less than a PAGE_SIZE no matter the start is zero or non-zero , so that Lustre can handle bellow sanity 317 test partial page write for a large PAGE_SIZE e.g. 64 KB and pass the test. That's the problem I want to solve.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> sanity.sh test_317() { ... 23836     # 23837     # sparse file test 23838     # Create file with a hole and write actual two blocks. Block count 23839     # must be 16. 23840     # 23841     dd <span class="code-keyword">if</span>=/dev/zero of=$DIR/$tfile bs=$grant_blk_size count=2 seek=5 \ 23842         conv=fsync || error <span class="code-quote">"Create file : $DIR/$tfile"</span> 23843 ... </pre> </div></div> <p>I am trying to understand all the details and  limitation including some mentioned by you e.g. RDMA partial page write, GPU direct write etc.</p> <p>I have a draft patch now which make client side send a niobuf,  which contains non-zero file start offset  and the real file end offset , to the server. This requires clip the page in the client side. And in the server side it only writes the necessary range(i.e. from the real non-zero file start offset to the file end offset).</p> <p>I will send the patch for review soon. Let's try if we can work out a solution.  Thanks.<br/>  <br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=paf0186" class="user-hover" rel="paf0186">paf0186</a> added a comment - 07/Nov/21 12:35 AM<br/> How do you handle the page cache?  Like, what's in there?  And how do you get the range for the clipping?  Etc.  Some of these questions will be answered with the patch, of course <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>But say you write this clipped partial page - What happens when you read it on the client which wrote it?  What is in the rest of the page?</p> <p>And, going on from there:<br/> What is in the rest of the page if the file was empty there?  And what is in the rest of the page if there was already data in the whole page when you write it?</p> <p>Basically what I am saying is unless you get <b>very</b> clever, this will break the page cache.</p> <p>You would also need to mark this page as non-mergable to avoid the RDMA issue, but that's easy to do.  The real sticking point is the page cache.<br/>  <br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=paf0186" class="user-hover" rel="paf0186">paf0186</a> added a comment - 07/Nov/21 12:38 AM<br/> One idea which Andreas and I had some time ago was the idea of something like marking the page as not up to date (this means <b>not</b> marking it as up to date, ie, the raw page state is not-up-to-date and up to date is a flag), so if the page was accessed, it would cause the client to re-read it from the server.</p> <p>This would mean the page was effectively uncached, which is a bit weird, but could work - I think the benefit is pretty limited since you can't easily combine these partial pages in to larger writes.  (RDMA issue again)</p> <p>But anyway, not setting written pages up to date turned out to be <b>really</b> complicated, and I decided it was unworkable.  The write code assumes pages are up to date as part writing them, and while I was able to work around a few things, I decided it felt like I was very much going against the intent of the code.<br/>  <br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=paf0186" class="user-hover" rel="paf0186">paf0186</a> added a comment - 07/Nov/21 12:42 AM<br/> We also need to ask:<br/> What's the benefit/what's the end goal, and how much do we have to do to get there?</p> <p>The benefit will be pretty limited if we can't also solve the RDMA issue.  The benefit would only apply for &lt; page writes, and each one would have to be sent to disk by itself.</p> <p>One way to solve the RDMA problem would be to send full pages over the network, but attach extra data in the RPC telling the server the actual range for each page.  This would be <b>very complicated</b>, I think, and involve new ways of handling writes on the client <b>and</b> server.</p> <p>And this assumes we can solve the page cache issue!<br/>  </p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=paf0186" class="user-hover" rel="paf0186">paf0186</a>, I am not understanding all the details yet. As <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a> said partial write is so complicated.</p> <p>But I will try my best to answer your questions. Jira is not convenient to discuss in threads. I will paste your words and answer them.</p> <p> </p> <p>------------------------------------------------------------------------------------------------------------------------------------------------------------------------ <br/> "<br/> More generally, about partial page i/o:<br/> Generally speaking, we can't have partial pages except at the start and end of each write<br/> "</p> <p>Yeah, I think that is the thing I want to accomplish. Partial write at the start and at the end of each write, maybe because page cache only records the start(offset) and the end(added by count).<br/> Each write issued by the user space is always a continuous range from the start to the end.<br/> Inside the Lustre the continuous range buf is converted into one extent to write. <br/> Lustre may combine multiple extents into one if they can be combined which means all the extents can become a continuous extent. Right?</p> <p> </p> <p>------------------------------------------------------------------------------------------------------------------------------------------------------------------------ <br/> "<br/> Patrick Farrell added a comment - 07/Nov/21 12:35 AM<br/> How do you handle the page cache?  Like, what's in there?  And how do you get the range for the clipping?  Etc.  Some of these questions will be answered with the patch, of course </p> <p>But say you write this clipped partial page - What happens when you read it on the client which wrote it?  What is in the rest of the page?</p> <p>And, going on from there:<br/> What is in the rest of the page if the file was empty there?  And what is in the rest of the page if there was already data in the whole page when you write it?</p> <p>Basically what I am saying is unless you get very clever, this will break the page cache.</p> <p>You would also need to mark this page as non-mergable to avoid the RDMA issue, but that's easy to do.  The real sticking point is the page cache.<br/> "</p> <p>Some questions maybe you can get answers from the patch. I will answer some of them. <br/> Before writing a partial page, if the page is already existing in the file the full page should be read from the disk so that we have an up to update page to write. If it is <br/> a new page than zero it or maybe no need to do anything. This is handled in the ll_prepare_partial_page().</p> <p> </p> <p>------------------------------------------------------------------------------------------------------------------------------------------------------------------------ <br/> "<br/> Patrick Farrell added a comment - 07/Nov/21 12:38 AM<br/> One idea which Andreas and I had some time ago was the idea of something like marking the page as not up to date (this means not marking it as up to date, ie, the raw page state is not-up-to-date and up to date is a flag), so if the page was accessed, it would cause the client to re-read it from the server.</p> <p>This would mean the page was effectively uncached, which is a bit weird, but could work - I think the benefit is pretty limited since you can't easily combine these partial pages in to larger writes.  (RDMA issue again)</p> <p>But anyway, not setting written pages up to date turned out to be really complicated, and I decided it was unworkable.  The write code assumes pages are up to date as part writing them, and while I was able to work around a few things, I decided it felt like I was very much going against the intent of the code.<br/> "</p> <p>If a partial page can not be combined, why not just leave it as a separate extent to write? This is the small read/write. As I know, ll_do_tiny_write() will handle a partial write which will update content of the range the user wants.<br/> Do you have some records about your work? Maybe I am not fully understanding your questions.<br/> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ </p> <p> </p> <p>"<br/> Patrick Farrell added a comment - 07/Nov/21 12:42 AM<br/> We also need to ask:<br/> What's the benefit/what's the end goal, and how much do we have to do to get there?</p> <p>The benefit will be pretty limited if we can't also solve the RDMA issue.  The benefit would only apply for &lt; page writes, and each one would have to be sent to disk by itself.</p> <p>One way to solve the RDMA problem would be to send full pages over the network, but attach extra data in the RPC telling the server the actual range for each page.  This would be very complicated, I think, and involve new ways of handling writes on the client and server.</p> <p>And this assumes we can solve the page cache issue!<br/> "<br/> The benefits I can imagine are:<br/> 1) Make small writing more efficient which maybe is a big drawback of Lustre I often heard. This drawback may become even worse for large PAGE_SIZE say 64K, this should be prove by testing;<br/> 2) Save disk space for large PAGE_SIZE say 64K, as now ldiskfs block allocation is aligned with PAGE_SIZE.</p> <p>.</p> <p> </p> <p>"xinliang &lt;xinliang.liu@linaro.org&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45569" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45569</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15223" title="Improve partial page read/write" class="issue-link" data-issue-key="LU-15223">LU-15223</a> ldiskfs: Make real partial page read/write<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 31f7077ce41dd69fdc5a3a916c330e094bdc9a32</p> Cloners LU-11667 Related LU-10300 Development Rank 1|i029vb: Rank (Obsolete) 9223372036854775807 Severity