https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11667 Lustre <p>sanity test 317 failed on ARM clients as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 317: Verify blocks get correctly update after truncate ================================ 15:30:27 (1542036627) 1+0 records in 1+0 records out 5242880 bytes (5.2 MB) copied, 0.467256 s, 11.2 MB/s /mnt/lustre/f317.sanity has size 2097152 OK /mnt/lustre/f317.sanity has size 4097 OK /mnt/lustre/f317.sanity has size 4000 OK /mnt/lustre/f317.sanity has size 509 OK /mnt/lustre/f317.sanity has size 0 OK 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0562888 s, 146 kB/s File: '/mnt/lustre/f317.sanity' Size: 24575 Blocks: 48 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115708605760525 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2018-11-12 15:30:29.000000000 +0000 Modify: 2018-11-12 15:30:29.000000000 +0000 Change: 2018-11-12 15:30:29.000000000 +0000 Birth: - sanity test_317: @@@@@@ FAIL: Expected Block 8 got 48 for f317.sanity </pre> </div></div> <p>Maloo report: <a href="https://testing.whamcloud.com/test_sets/074afc02-e7bf-11e8-815b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/074afc02-e7bf-11e8-815b-52540065bddc</a></p> Arch: aarch64 (client) LU-11667

sanity test 317: FAIL: Expected Block 8 got 48 for f317.sanity

Bug Minor Resolved Fixed WC Triage Jian Yu arm arm-server ppc Wed, 14 Nov 2018 09:07:21 +0000 Sat, 10 Dec 2022 04:19:09 +0000 Sat, 20 Nov 2021 17:55:31 +0000 Lustre 2.12.0 Lustre 2.12.4 Lustre 2.15.0 0 8 <p>Jian Yu (yujian@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33656" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33656</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11667" title="sanity test 317: FAIL: Expected Block 8 got 48 for f317.sanity" class="issue-link" data-issue-key="LU-11667"><del>LU-11667</del></a> tests: disable sanity test 317 for ARM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4f6c573ff0f3ecd69a835a19d8402a69f39d088e</p> <p>This could just be a test defect, because the <tt>dd bs=$grant_blk_size count=2 seek=5</tt> can write chunks that are not aligned on a PAGE_SIZE boundary if blocksize != PAGE_SIZE.</p> <p>We're seeing the same failure for PPC client testing. See <a href="https://testing.whamcloud.com/test_sets/6b8903ee-4d49-11ea-b58e-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/6b8903ee-4d49-11ea-b58e-52540065bddc</a> for logs. </p> <p>Created two same size 10B  file in home dir and /mnt/lustre dir, if the backend filesystem block size is 4K. Then the inode allocated blocks should be the same( that is 8 if count by block size 512B).</p> <p>Test file created at home dir:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ getconf PAGESIZE 65536 $ echo <span class="code-quote">"123456789"</span> &gt; ~/testfile $ stat ~/testfile File: /root/testfile Size: 10 Blocks: 8 IO Block: 65536 regular file Device: fc02h/64514d Inode: 12863429 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2021-09-16 02:51:21.268641287 +0000 Modify: 2021-09-16 03:08:05.382557951 +0000 Change: 2021-09-16 03:08:05.382557951 +0000 Birth: - $ stat -c %b ~/testfile 8 $ stat -c %B ~/testfile 512 $ stat -c %s ~/testfile 10 $ stat -f ~/testfile File: <span class="code-quote">"/root/testfile"</span> ID: fc0200000000 Namelen: 255 Type: xfs Block size: 4096 Fundamental block size: 4096 Blocks: Total: 52272379 Free: 45840170 Available: 45840170 Inodes: Total: 104549824 Free: 104176363 </pre> </div></div> <p>Test file created at Lustre dir:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ getconf PAGESIZE 65536 $ echo <span class="code-quote">"123456789"</span> &gt; /mnt/lustre/testfile $ stat -c %s /mnt/lustre/testfile 10 $ stat -c %B /mnt/lustre/testfile 512 $ stat -c %b /mnt/lustre/testfile 128 $ stat /mnt/lustre/testfile File: /mnt/lustre/testfile Size: 10 Blocks: 128 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272502274 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2021-09-16 02:53:57.000000000 +0000 Modify: 2021-09-16 03:07:28.000000000 +0000 Change: 2021-09-16 03:07:28.000000000 +0000 Birth: - $ stat -f /mnt/lustre/testfile File: <span class="code-quote">"/mnt/lustre/testfile"</span> ID: 2c54f96600000000 Namelen: 255 Type: lustre Block size: 4096 Fundamental block size: 4096 Blocks: Total: 78276 Free: 77931 Available: 71141 Inodes: Total: 100000 Free: 99726 </pre> </div></div> <p>But the Lustre test file's inode blocks is 128. This should be wrong?</p> <p> </p> <p> </p> <p>I think this has always been the case for writes from 64KB PAGE_SIZE clients (e.g. back to ia64). The reason is that the client sends a full-page write, because it is only tracking dirty pages, and the server writes the full amount of data sent by the client. I suspect that ext4 is handling this by having multiple 4KB buffer_heads on a 64KB page, and using the buffer dirty state to determine which pages to write, but Lustre doesn't use buffer heads.</p> <p>HI Andreas,</p> <p>I found that this issue happens at Arm 64K PAGE_SIZE OST server.</p> <p>When create a file, blocks are allocated with PAGE_SIZE aligned, see function osd_ldiskfs_map_inode_pages().</p> <p> E.g. for 64K PAGE_SIZE Arm64 OST server, if create a file with size less than 64K, it actually allocates 128 blocks each block 512 Bytes.</p> <p>We need to adjust the test for 64K PAGE_SIZE OST server.</p> <p>I am thinking if we should make blocks allocation aligned with BLOCK_SIZE as ext4, which could save space for large PAGE_SIZE e.g. 64K. Then no need to make change to the test case. And I have a look at the code it seems both OSC client  and OST server need to adjust for this. The client always sends no hole pages (currently page start offset is always 0) to the server for writing now. And the server side needs to adjust making blocks allocation aligned with block size. </p> <p>Find out the client side page clip related code:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 298 <span class="code-object">int</span> osc_io_commit_async(<span class="code-keyword">const</span> struct lu_env *env, 299 <span class="code-keyword">const</span> struct cl_io_slice *ios, 300 struct cl_page_list *qin, <span class="code-object">int</span> from, <span class="code-object">int</span> to, 301 cl_commit_cbt cb) 302 { ... 315 <span class="code-comment">/* Handle partial page cases */</span> 316 last_page = cl_page_list_last(qin); 317 <span class="code-keyword">if</span> (oio-&gt;oi_lockless) { 318 page = cl_page_list_first(qin); 319 <span class="code-keyword">if</span> (page == last_page) { 320 cl_page_clip(env, page, from, to); 321 } <span class="code-keyword">else</span> { 322 <span class="code-keyword">if</span> (from != 0) 323 cl_page_clip(env, page, from, PAGE_SIZE); 324 <span class="code-keyword">if</span> (to != PAGE_SIZE) 325 cl_page_clip(env, last_page, 0, to); 326 } 327 } 328 329 ll_pagevec_init(pvec, 0); </pre> </div></div> <p>Currently, it seems a normal write don't go into this "if (oio-&gt;oi_lockless) {" part code. Anyone know why it is oi_lockless? @<a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="external-link" rel="nofollow">Andreas Dilger</a></p> <p> </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=paf0186" class="user-hover" rel="paf0186">paf0186</a> you are probably the most interested in changing this code. Handling sub-page writes for ARM would not be very different than sub-page writes for x86, which would potentially allow eg. IO500 unaligned writes to be handled much more efficiently. </p> <p>Xinliang,</p> <p>I am not 100% sure I understand your question - Are you saying it <b>is</b> oi_lockless?  It should not be.  This (commit_async) code is buffered, and lockless buffered is broken and also off by default.  I have a patch to remove it, but it's normally off anyway.</p> <p>What are you looking for/hoping for here?</p> <p>Note we clip pages in other places too.</p> <p>"I am thinking if we should make blocks allocation aligned with BLOCK_SIZE as ext4, which could save space for large PAGE_SIZE e.g. 64K. Then no need to make change to the test case. And I have a look at the code it seems both OSC client  and OST server need to adjust for this. The client always sends no hole pages (currently page start offset is always 0) to the server for writing now. And the server side needs to adjust making blocks allocation aligned with block size. <br/>  "</p> <p>Can you talk more about what you're thinking?  I am not quite what the implication of changing block allocation on the server would be on the client.  Why does changing server block allocation filter back to the client like this?</p> <p>More generally, about partial page i/o:<br/> Generally speaking, we can't have partial pages except at the start and end of each write - that's a limitation of infiniband, but there are also page cache restrictions.</p> <p>In general, RDMA can be unaligned at the start, and unaligned at the end, but that's it.  This applies even when combining multiple RDMA regions - it's some limitation of the hardware/drivers.  So we have a truly unaligned I/O(with a partial page at beginning and end), but then we can't combine it with other I/Os.</p> <p>There is also a page cache limitation here.  The Linux page cache insists on working with full pages - It will only allow partial pages at file_size.  So, eg, a 3K file is a single page with 3K in it, and we can write just 3K.  But if we want to write 3K in to a large 'hole' in a file, Linux will enforce writing PAGE_SIZE.  This is not a restriction we can easily remove, it is an important part of the page cache.</p> <p>By the way, I am happy to keep talking about this, if you have thoughts or questions or whatever.  I've looked at sub-page I/O a few times, but you may have a different idea than what I have tried.</p> <p>Patrick, I was thinking that if we can handle a write (uncached) from the client that is RDMA 64KB, but has a non-zero start and end offset (4KB initially), it might be generalizable to any byte offset. </p> <p>I'm aware of the RDMA limitations, but I'm wondering if those can be bypassed (if necessary) by transferring a whole page over the network, but store it into a temporary page and copy the data for a cached/unaligned read-modify-write on the server to properly align the data. The content of the start/end of the page sent from the client would be irrelevant, since it will be trimmed by the server anyway when the copy is done</p> <p> While the copy might be expensive for very large writes, my expectation is that this would be most useful for small writes. That does raise the question of whether the data could be transferred in the RPC as a short write, but for GPU direct we require RDMA to send the data directly from the GPU RAM to the OSS. Maybe it is just a matter of generalizing the short write handling to allow copying from the middle of an RDMA page?</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=paf0186" class="user-hover" rel="paf0186">paf0186</a> and <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="external-link" rel="nofollow">Andreas Dilger</a>, thank you for the clarification about partial page write. It really helps me a lot.</p> <p>For ldiskfs backend filesystem,  I see that if the user issue a partial page cached write the Lustre (including client side and server side) will convert it in to a full page write. I want to make Lustre do a real partial page write inside which with the length less than a PAGE_SIZE no matter the start is zero or non-zero , so that Lustre can handle bellow sanity 317 test partial page write for a large PAGE_SIZE e.g. 64 KB and pass the test. That's the problem I want to solve.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> sanity.sh test_317() { ... 23836     # 23837     # sparse file test 23838     # Create file with a hole and write actual two blocks. Block count 23839     # must be 16. 23840     # 23841     dd <span class="code-keyword">if</span>=/dev/zero of=$DIR/$tfile bs=$grant_blk_size count=2 seek=5 \ 23842         conv=fsync || error <span class="code-quote">"Create file : $DIR/$tfile"</span> 23843 ... </pre> </div></div> <p>I am trying to understand all the details and  limitation including some mentioned by you e.g. RDMA partial page write, GPU direct write etc.</p> <p>I have a draft patch now which make client side send a niobuf,  which contains non-zero file start offset  and the real file end offset , to the server. This requires clip the page in the client side. And in the server side it only writes the necessary range(i.e. from the real non-zero file start offset to the file end offset).</p> <p>I will send the patch for review soon. Let's try if we can work out a solution.  Thanks.</p> <p>How do you handle the page cache?  Like, what's in there?  And how do you get the range for the clipping?  Etc.  Some of these questions will be answered with the patch, of course <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>But say you write this clipped partial page - What happens when you read it on the client which wrote it?  What is in the rest of the page?</p> <p>And, going on from there:<br/> What is in the rest of the page if the file was empty there?  And what is in the rest of the page if there was already data in the whole page when you write it?</p> <p>Basically what I am saying is unless you get <b>very</b> clever, this will break the page cache.</p> <p>You would also need to mark this page as non-mergable to avoid the RDMA issue, but that's easy to do.  The real sticking point is the page cache.</p> <p>One idea which Andreas and I had some time ago was the idea of something like marking the page as not up to date (this means <b>not</b> marking it as up to date, ie, the raw page state is not-up-to-date and up to date is a flag), so if the page was accessed, it would cause the client to re-read it from the server.</p> <p>This would mean the page was effectively uncached, which is a bit weird, but could work - I think the benefit is pretty limited since you can't easily combine these partial pages in to larger writes.  (RDMA issue again)</p> <p>But anyway, not setting written pages up to date turned out to be <b>really</b> complicated, and I decided it was unworkable.  The write code assumes pages are up to date as part writing them, and while I was able to work around a few things, I decided it felt like I was very much going against the intent of the code.</p> <p>We also need to ask:<br/> What's the benefit/what's the end goal, and how much do we have to do to get there?</p> <p>The benefit will be pretty limited if we can't also solve the RDMA issue.  The benefit would only apply for &lt; page writes, and each one would have to be sent to disk by itself.</p> <p>One way to solve the RDMA problem would be to send full pages over the network, but attach extra data in the RPC telling the server the actual range for each page.  This would be <b>very complicated</b>, I think, and involve new ways of handling writes on the client <b>and</b> server.</p> <p>And this assumes we can solve the page cache issue!</p> <p> </p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=paf0186" class="user-hover" rel="paf0186">paf0186</a> and  <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a>,  As partial write is so complicated we might take a long time for it. let's create another Jira card for partial write and discuss there? I also send a draft patch for review.</p> <p><a href="https://jira.whamcloud.com/browse/LU-15223" title="Improve partial page read/write" class="issue-link" data-issue-key="LU-15223">LU-15223</a></p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45395/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45395/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11667" title="sanity test 317: FAIL: Expected Block 8 got 48 for f317.sanity" class="issue-link" data-issue-key="LU-11667"><del>LU-11667</del></a> tests: Fix sanity test 317 for 64K PAGE_SIZE OST<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 63d4d9ff2f5c8cc992ca6b2f698bb43a3257bfb3</p> <p>Work around landed. Proper fix is being done in <a href="https://jira.whamcloud.com/browse/LU-15223" title="Improve partial page read/write" class="issue-link" data-issue-key="LU-15223">LU-15223</a></p> Cloners LU-15223 Duplicate Related LU-10300 Development Rank 1|i006db: Rank (Obsolete) 9223372036854775807 Severity