https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11596 Lustre <p>sanity test 42d failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.lustre-*.cur_*grant_bytes Lustre: DEBUG MARKER: /usr/sbin/lctl get_param osc.lustre-*.cur_*_bytes Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity test_42d: @@@@@@ FAIL: failed: client:143294464 server: 143110144. Lustre: DEBUG MARKER: sanity test_42d: @@@@@@ FAIL: failed: client:143294464 server: 143110144. </pre> </div></div> <p>Dmesg on OSS node:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == sanity test 42d: test complete truncate of file with cached dirty data ============================ 01:34:08 (1540517648) Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2&gt;/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl get_param obdfilter.lustre-OST*.{tot_granted,tot_pending,grant_precreate} Lustre: DEBUG MARKER: /usr/sbin/lctl get_param obdfilter.lustre-OST*.tot* obdfilter.lustre-OST*.grant_* Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity test_42d: @@@@@@ FAIL: failed: client:143294464 server: 143110144. Lustre: DEBUG MARKER: sanity test_42d: @@@@@@ FAIL: failed: client:143294464 server: 143110144. </pre> </div></div> <p>Maloo report: <a href="https://testing.whamcloud.com/test_sets/88bbf5c2-d9d0-11e8-b46b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/88bbf5c2-d9d0-11e8-b46b-52540065bddc</a></p> <p>sanity tests 63a, 63b, 64a, 64b, 64c also failed with the same issue.</p> Lustre Build: <a href="https://build.whamcloud.com/job/lustre-master/3811">https://build.whamcloud.com/job/lustre-master/3811</a> <br/> Distro/Arch: RHEL7.5/aarch64 (client), RHEL7.5/x86_64 (server) LU-11596

sanity test_42d/test_42e: FAIL: failed: client:143294464 server: 143110144

Bug Major Resolved Fixed James A Simmons Jian Yu arm ppc rhel8 Thu, 1 Nov 2018 17:40:46 +0000 Tue, 21 Mar 2023 20:58:40 +0000 Tue, 18 Jan 2022 14:20:07 +0000 Lustre 2.12.0 Lustre 2.13.0 Lustre 2.14.0 Lustre 2.12.4 Lustre 2.12.5 Lustre 2.12.6 Lustre 2.15.0 0 9 <p>Jian Yu (yujian@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33652" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33652</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11596" title="sanity test_42d/test_42e: FAIL: failed: client:143294464 server: 143110144" class="issue-link" data-issue-key="LU-11596"><del>LU-11596</del></a> tests: disable sanity tests 42d,63<span class="error">&#91;a,b&#93;</span>,64<span class="error">&#91;a-c&#93;</span> for ARM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f03165045efacafb69204d460c3409ab91f00a3a</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33652/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33652/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11596" title="sanity test_42d/test_42e: FAIL: failed: client:143294464 server: 143110144" class="issue-link" data-issue-key="LU-11596"><del>LU-11596</del></a> tests: disable several sanity sub-tests for ARM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ef38bdfd2b7e672e1886d5c9874e4749ce2ac258</p> <p>It looks like this is one of the few test failures that seems to legitimately be caused by the code running on ARM 64KB PAGE_SIZE. I suspect that there is somewhere in the client (OSC) or server (OFD) grant code that is assuming <tt>PAGE_SIZE</tt> = 4096, and is adding/subtracting 4096 bytes instead of <tt>PAGE_SIZE</tt>. In the case reported in the bug title, the difference between the grant values is 184320 bytes, or 45x 4096 bytes.</p> <p>Given that this is a 100% failure it should be fairly straight forward to debug. Note, however, that it is possible the grant has become out-of-sync in some earlier test, and these tests are failing because they are the only ones in <tt>$GRANT_CHECK_LIST</tt>. It may be useful to modify <tt>test-framework.c::check_grant()</tt> to allow setting an environment variable (e.g. <tt>GRANT_CHECK_ALWAYS=yes</tt>) to check grant for every subtest to catch the actual source of the problem.</p> <p>Thank you Andreas for the suggestions. Let me look into the grant codes.</p> <p>Faced the same problem on arm session for another sanity tests - 63a, 63b, 64a, 64c<br/> <a href="https://testing.whamcloud.com/test_sets/f33166da-ecdb-11e8-adf2-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f33166da-ecdb-11e8-adf2-52540065bddc</a></p> <blockquote> <p>Faced the same problem on arm session for another sanity tests - 63a, 63b, 64a, 64c<br/> <a href="https://testing.whamcloud.com/test_sets/f33166da-ecdb-11e8-adf2-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f33166da-ecdb-11e8-adf2-52540065bddc</a></p></blockquote> <p>Sergey,<br/> if you are still hitting these failures on ARM, then it means your patch is not updated to the latest master that includes the above patch to skip these tests. You could rebase your patch to get these fixes, but it is not critical for review-ldiskfs-arm because that test session is optional and does not affect the Maloo +/-1 result.</p> <p> </p> <p>It looks like there are also failures on sanity test_64d on ARM, so that should be added to the skip list:<br/> <a href="https://testing.whamcloud.com/test_sets/48e911d6-2a08-11e9-b901-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/48e911d6-2a08-11e9-b901-52540065bddc</a></p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34432" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34432</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11596" title="sanity test_42d/test_42e: FAIL: failed: client:143294464 server: 143110144" class="issue-link" data-issue-key="LU-11596"><del>LU-11596</del></a> tests: skip sanity test_64d for ARM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 74d6824790616f3a997783b65d3c5e0af4a1566e</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34432/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34432/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11596" title="sanity test_42d/test_42e: FAIL: failed: client:143294464 server: 143110144" class="issue-link" data-issue-key="LU-11596"><del>LU-11596</del></a> tests: skip sanity test_64d for ARM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 946764287309a274fa805d930e4f1126d4a66570</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37463" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37463</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11596" title="sanity test_42d/test_42e: FAIL: failed: client:143294464 server: 143110144" class="issue-link" data-issue-key="LU-11596"><del>LU-11596</del></a> tests: skip sanity tests for PPC<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1916b688570501112977b7062c3affcf13a6aaaa</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37876" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37876</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11596" title="sanity test_42d/test_42e: FAIL: failed: client:143294464 server: 143110144" class="issue-link" data-issue-key="LU-11596"><del>LU-11596</del></a> tests: wait for grant to stop changing<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fd1d4176367a33eccd2f0715de8e0abf4fcd4996</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37876/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37876/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11596" title="sanity test_42d/test_42e: FAIL: failed: client:143294464 server: 143110144" class="issue-link" data-issue-key="LU-11596"><del>LU-11596</del></a> tests: wait for grant to stop changing<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 75ebfa0b924a41d15aea026dc9a29bec39d1419a</p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/40758" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40758</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11596" title="sanity test_42d/test_42e: FAIL: failed: client:143294464 server: 143110144" class="issue-link" data-issue-key="LU-11596"><del>LU-11596</del></a> tests: re-enable sanity grant test for ARM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b11e62e18bbdb56102f872c8c3aeefb2faa0e799</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="external-link" rel="nofollow">Andreas Dilger</a>'s guess was right. I can make the test pass on arm CentOS8. When changing the write amount to page size align:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> @@ -5091,7 +5091,7 @@ test_42a() { sync; sleep 1; sync # just to be safe BEFOREWRITES=`count_ost_writes` lctl get_param -n osc.*[oO][sS][cC][_-]*.cur_grant_bytes | grep <span class="code-quote">"[0-9]"</span> - dd <span class="code-keyword">if</span>=/dev/zero of=$DIR/f42a bs=1024 count=100 + dd <span class="code-keyword">if</span>=/dev/zero of=$DIR/f42a bs=$PAGE_SIZE count=2 </pre> </div></div> <p>And looking into the client end grant calculation code, it likely related to this part code:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 830 <span class="code-object">int</span> osc_extent_finish(<span class="code-keyword">const</span> struct lu_env *env, struct osc_extent *ext, 831 <span class="code-object">int</span> sent, <span class="code-object">int</span> rc) 832 { ... 865 } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (blocksize &lt; PAGE_SIZE &amp;&amp; 866 last_count != PAGE_SIZE) { 867 /* For <span class="code-object">short</span> writes we shouldn't count parts of pages that 868 * span a whole chunk on the OST side, or our accounting goes 869 * wrong. Should match the code in filter_grant_check. */ 870 <span class="code-object">int</span> offset = last_off &amp; ~PAGE_MASK; 871 <span class="code-object">int</span> count = last_count + (offset &amp; (blocksize - 1)); 872 <span class="code-object">int</span> end = (offset + last_count) &amp; (blocksize - 1); 873 <span class="code-keyword">if</span> (end) 874 count += blocksize - end; 875 876 lost_grant = PAGE_SIZE - count; 877 } 878 <span class="code-keyword">if</span> (ext-&gt;oe_grants &gt; 0) 879 osc_free_grant(cli, nr_pages, lost_grant, ext-&gt;oe_grants); </pre> </div></div> <p>This part code will never go into on x86_64 as page size and block size are always 4k. But this part code looks doesn't match the sever end related code of tgt_grant_check() on Arm RHEL/CentOS, the blocksize seems always 4k, but  page size is 64k on Arm RHEL/CentOS:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 645 <span class="code-keyword">static</span> inline u64 tgt_grant_rnb_size(struct obd_export *exp, 646 struct lu_target *lut, 647 struct niobuf_remote *rnb) 648 { ... 659 660 /* The network buffer might span several blocks, align it on block 661 * boundaries */ 662 bytes = rnb-&gt;rnb_offset &amp; (blksize - 1); 663 bytes += rnb-&gt;rnb_len; 664 end = bytes &amp; (blksize - 1); 665 <span class="code-keyword">if</span> (end) 666 bytes += blksize - end; </pre> </div></div> <p> </p> <p>But I have no idea how to fix now. Please give me some help, thanks.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=xinliang" class="user-hover" rel="xinliang">xinliang</a>, thanks for looking into this. Strictly speaking, the grant is related to the amount of space consumed in the back-end filesystem, which is usually in 4KB multiples (blocksize). The reason that PAGE_SIZE is involved is because the client VM cannot track cached dirty data of less than a full page. As a result, the 64KB clients will need to send 64KB writes, even though only a byte of data was modified in the page. This will consume more of the client grant to dirty the page, but if there is still free space in the OST it will receive replacement grant in the RPC reply.</p> <blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Should match the code in filter_grant_check. </pre> </div></div></blockquote> <p>The <tt>filter_grant_check()</tt> function doesn't exist anymore, and was replaced by <tt>tgt_grant_check()</tt>, as you mentioned. If you are updating this code, please also fix this comment.</p> <p>It looks like the server code is more focussed on handling the case of request size &lt; blocksize (ZFS), compared to request size &gt; blocksize (the common case). Where this <em>might</em> be going wrong is writes at the end of the file, since <tt>osc_extent_finish()</tt> is not consuming a whole page worth of grant if the end of file is 4KB * N less than the full PAGE_SIZE?</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="external-link" rel="nofollow">Andreas Dilger</a>, thanks for the explanation about grant, will update the comment when send the fixed patch.</p> <p>After looking into the code and git log, I found that starting from this commit (bd1e41672c <a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a> grant: add support for OBD_CONNECT_GRANT_PARAM), this OSC part code,</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 830 <span class="code-object">int</span> osc_extent_finish(<span class="code-keyword">const</span> struct lu_env *env, struct osc_extent *ext, 831 <span class="code-object">int</span> sent, <span class="code-object">int</span> rc) 832 { ... 865 } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (blocksize &lt; PAGE_SIZE &amp;&amp; 866 last_count != PAGE_SIZE) { 867 /* For <span class="code-object">short</span> writes we shouldn't count parts of pages that 868 * span a whole chunk on the OST side, or our accounting goes 869 * wrong. Should match the code in filter_grant_check. */ 870 <span class="code-object">int</span> offset = last_off &amp; ~PAGE_MASK; 871 <span class="code-object">int</span> count = last_count + (offset &amp; (blocksize - 1)); 872 <span class="code-object">int</span> end = (offset + last_count) &amp; (blocksize - 1); 873 <span class="code-keyword">if</span> (end) 874 count += blocksize - end; 875 876 lost_grant = PAGE_SIZE - count; 877 } 878 <span class="code-keyword">if</span> (ext-&gt;oe_grants &gt; 0) 879 osc_free_grant(cli, nr_pages, lost_grant, ext-&gt;oe_grants); </pre> </div></div> <p>begins to mismatch the code of OST tgt_grant_check</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 708 <span class="code-keyword">static</span> void tgt_grant_check(<span class="code-keyword">const</span> struct lu_env *env, struct obd_export *exp, 709 struct obdo *oa, struct niobuf_remote *rnb, 710 <span class="code-object">int</span> niocount, u64 *left) 711 { ... 716 unsigned <span class="code-object">long</span> ungranted = 0; 717 unsigned <span class="code-object">long</span> granted = 0; ... 740 } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (exp_grant_param_supp(exp) &amp;&amp; oa-&gt;o_grant_used &gt; 0) { 741 /* Client supports the <span class="code-keyword">new</span> grant parameters and is telling us 742 * how much grant space it consumed <span class="code-keyword">for</span> <span class="code-keyword">this</span> bulk write. 743 * Although all rnbs are supposed to have the OBD_BRW_FROM_GRANT 744 * flag set, we will scan the rnb list and looks <span class="code-keyword">for</span> non-cache 745 * I/O in <span class="code-keyword">case</span> it changes in the <span class="code-keyword">future</span> */ 746 <span class="code-keyword">if</span> (ted-&gt;ted_grant &gt;= oa-&gt;o_grant_used) { 747 /* skip grant accounting <span class="code-keyword">for</span> rnbs with 748 * OBD_BRW_FROM_GRANT and just used grant consumption 749 * claimed in the request */ 750 granted = oa-&gt;o_grant_used; 751 skip = <span class="code-keyword">true</span>; 752 } <span class="code-keyword">else</span> { ... 773 } 774 775 <span class="code-keyword">for</span> (i = 0; i &lt; niocount; i++) { 776 <span class="code-object">int</span> bytes; 777 778 <span class="code-keyword">if</span> ((rnb[i].rnb_flags &amp; OBD_BRW_FROM_GRANT)) { 779 <span class="code-keyword">if</span> (skip) { 780 rnb[i].rnb_flags |= OBD_BRW_GRANTED; 781 <span class="code-keyword">continue</span>; 782 } ... 829 } 830 831 /* record in o_grant_used the actual space reserved <span class="code-keyword">for</span> the I/O, will be 832 * used later in tgt_grant_commmit() */ 833 oa-&gt;o_grant_used = granted + ungranted; ... 872 } </pre> </div></div> <p> </p> <p>So when client PAGE_SIZE is 64KB bigger than backend file system block size 4KB (Ldiskfs),</p> <p>The matched thing should be:</p> <p>If both server and client support OBD_CONNECT_GRANT_PARAM, the lost_grant in osc_extent_finish should be 0, because server will use grant consumption claimed  by the client. </p> <p>And note that, in this case, lost_grant = (client claim o_grant_used) - (OST actual+ o_grant_used)  = 0.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/40758/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40758/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11596" title="sanity test_42d/test_42e: FAIL: failed: client:143294464 server: 143110144" class="issue-link" data-issue-key="LU-11596"><del>LU-11596</del></a> osc: Fix and re-enable sanity grant test for ARM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7d3edce0650f0b66bdd280373a54a16cb28b8469</p> Duplicate LU-10033 LU-10571 LU-14630 Related LU-12255 LU-13838 LU-10300 Development Rank 1|i005hj: Rank (Obsolete) 9223372036854775807 Severity