https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15748 Lustre <p>This issue was created by maloo for Cliff White &lt;cwhite@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/1c15b404-f4af-4275-9a77-f5a50b30ca56" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1c15b404-f4af-4275-9a77-f5a50b30ca56</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>keep default fallocate mode: 0 fallocate failed, error Operation not supported, mode 0, offset 62914560, len 4194304 sanity test_150b: @@@@@@ FAIL: fallocate failed Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:6273:error() = /usr/lib64/lustre/tests/sanity.sh:13441:test_150b() = /usr/lib64/lustre/tests/test-framework.sh:6576:run_one() = /usr/lib64/lustre/tests/test-framework.sh:6623:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:6465:run_test() = /usr/lib64/lustre/tests/sanity.sh:13443:main() Dumping lctl log to /autotest/autotest-1/2022-04-03/lustre-master_full-part-2_4283_1_29_b51025b5-001f-45f2-9eb2-ebe47692e36c//sanity.test_150b.*.1649017567.log </pre> </div></div> LU-15748

interop: sanity test_150b: fallocate failed, error Operation not supported, mode 0, offset 62914560, len 4194304

Bug Major Resolved Fixed Arshad Hussain Maloo Thu, 14 Apr 2022 20:50:22 +0000 Thu, 27 Apr 2023 15:48:27 +0000 Tue, 29 Nov 2022 14:04:27 +0000 Lustre 2.15.0 Lustre 2.15.0 0 7 <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=arshad512" class="user-hover" rel="arshad512">arshad512</a> , are you able to take a look at this?</p> <p>@Patrick Farrell, sorry, I missed this JIRA. I will have a look and upadate on this.</p> <p> </p> <p> </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=arshad512" class="user-hover" rel="arshad512">arshad512</a> - no worries, it's quite new.  Thanks for taking a look.</p> <p>@Andreas, @Patrick</p> <p>This is an inter-op bug. Seen on 2.14 client and 2.15.0-RC3 (server). New check added under lustre/ofd/ofd_dev.c when fallocate(punch) was introduced. And was<br/> always failing. This is because, The o_valid field of obdo was not setting correct flags<br/> when passing it to server under osc.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>if ((oa-&gt;o_valid &amp; (OBD_MD_FLSIZE | OBD_MD_FLBLOCKS)) !=                     (OBD_MD_FLSIZE | OBD_MD_FLBLOCKS)) {      ... }</pre> </div></div> <p>This was reproducable as (without patch) on 2.14 client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ./check_fallocate /mnt/lustre/a fallocate failed, error Operation not supported, mode 0, offset 62914560, len 4194304</pre> </div></div> <p> </p> <p>After patch on 2.14 client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ./check_fallocate /mnt/lustre/a # ls -ali /mnt/lustre/a  144115205272502273 -rwx------ 1 root root 125829120 Apr 19 05:56 /mnt/lustre/a</pre> </div></div> <p>Since the check is valid. The patch must be back-ported to 2.14.  Other thing to check is that the within check_fallocate.c the function test_prealloc_nonsparse() always carries correct o_valid flags. However, test_prealloc_sparse() does not and which was failing.</p> <p><del>"Arshad Hussain &lt;arshad.hussain@aeoncomputing.com&gt;" uploaded a new patch:</del> <a href="https://review.whamcloud.com/47091" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47091</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-15748" title="interop: sanity test_150b: fallocate failed, error Operation not supported, mode 0, offset 62914560, len 4194304" class="issue-link" data-issue-key="LU-15748"><del>LU-15748</del></a> osc: Correctly pass obdo(o_valid) flags</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: b2_14</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: 3dd05b38f5a3e779dea73b824ad3a6390e387327</del></p> <p>It isn't really possible to retroactively fix older releases in this way. Even if this patch landed on b2_14 it would not help the sites that are running 2.14.0.</p> <p>Do you know if 2.14.0 correctly populated these fields, but just didn't set the flags? In that case, one solution would be to disable the new flag check on master <b>before</b> 2.15.0 is released, and put it under:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> #ifdef LUSTRE_BUILD_VERSION &gt; OBD_OCD_VERSION(2, 18, 52, 0) </pre> </div></div> <p>and only check that the <b>values</b> are set for the next few releases. Also, we might (initially) make this check conditional on the use of <tt>FALLOC_FL_PUNCH_HOLE</tt>, since that is when the client started setting these flags.</p> <p>Alternately, the server could check if the client version is &gt;= 2.15.0 before enforcing the check, but this makes things complicated if backporting the punch feature to older releases (though at worst a sanity check is removed that might still be reasonably validated by checking the <tt>o_size</tt> and <tt>o_blocks</tt> values are sane. Alternately, it could be checked by an <tt>OBD_CONNECT2_*</tt> flag already added in 2.15.0, but I'm a bit reluctant to burn a new flag for this minor issue (though it wouldn't be the end of the world if others think that is needed).</p> <p>It looks like the <b>lack</b> of <tt>OBD_CONNECT_TRUNCLOCK</tt> would indicate a newer client (it was set by 2.14.0 clients and sent to both MDTs and OSTs, but not by 2.15.0 clients), which would give us something like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* was OBD_CONNECT_TRUNCLOCK           0x400ULL *locks on server <span class="code-keyword">for</span> punch */</span> <span class="code-comment">/* temporary usage until 2.21.53 to indicate pre-2.15 client, see LU-15478 */</span> #define OBD_CONNECT_OLD_FALLOC 0x400ULL <span class="code-comment">/* missing o_valid flags */</span> /* * fallocate() start and end are passed in o_size and o_blocks * on the wire. Clients 2.15.0 and newer should always set * the OBD_MD_FLSIZE and OBD_MD_FLBLOCKS valid flags, but some * older client versions did not. We permit older clients to * not set these flags, checking their version by proxy using * the lack of OBD_CONNECT_TRUNCLOCK to imply 2.14.0 and older. * * Return -EOPNOTSUPP to also work with older clients not * supporting newer server modes. */ <span class="code-keyword">if</span> ((oa-&gt;o_valid &amp; (OBD_MD_FLSIZE | OBD_MD_FLBLOCKS)) != (OBD_MD_FLSIZE | OBD_MD_FLBLOCKS) #<span class="code-keyword">if</span> LUSTRE_VERSION_CODE &lt; OBD_OCD_VERSION(2, 21, 53, 0)            &amp;&amp; !(tgt_conn_flags(tsi) &amp; OBD_CONNECT_OLD_FALLOC) #endif ) RETURN(-EOPNOTSUPP); start = oa-&gt;o_size; end = oa-&gt;o_blocks; <span class="code-comment">/* verify arguments are sane (len &lt;= 0 also denied by client VFS) */</span> <span class="code-keyword">if</span> (start &gt;= end) RETURN(-EINVAL); /* * mode == 0 (which is standard prealloc) and PUNCH is supported * Rest of mode options are not supported yet. */ mode = oa-&gt;o_falloc_mode; <span class="code-keyword">if</span> (mode &amp; ~(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE)) RETURN(-EOPNOTSUPP); </pre> </div></div> <p><b>NOTE</b> the same problem exists in <tt>mdt_fallocate_hdl()</tt> but has not even been fixed with the <a href="https://jira.whamcloud.com/browse/LU-15551" title="Interop: Various tests fail with fallocate “Protocol error” errors" class="issue-link" data-issue-key="LU-15551"><del>LU-15551</del></a> patch. It will need a similar change to the logic.</p> <p>"Arshad Hussain &lt;arshad.hussain@aeoncomputing.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47098" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47098</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15748" title="interop: sanity test_150b: fallocate failed, error Operation not supported, mode 0, offset 62914560, len 4194304" class="issue-link" data-issue-key="LU-15748"><del>LU-15748</del></a> osc: Make older clients work with newer servers without setting OBD_MD_* flags<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 22addaa242320addb7140c607a68f3e8500e8cc2</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47098/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47098/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15748" title="interop: sanity test_150b: fallocate failed, error Operation not supported, mode 0, offset 62914560, len 4194304" class="issue-link" data-issue-key="LU-15748"><del>LU-15748</del></a> osc: fallocate interop for 2.14 clients<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 79053592966792a21912390b9db09b1ab1d61e2a</p> <p>Landed for 2.15</p> <p>I think the patch to fix the interop is backward and needs a minor fix.</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47548" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47548</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15748" title="interop: sanity test_150b: fallocate failed, error Operation not supported, mode 0, offset 62914560, len 4194304" class="issue-link" data-issue-key="LU-15748"><del>LU-15748</del></a> ofd: fix fallocate interop for older clients<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 98529e39d92e9d46e98b015327d52d0e382481b8</p> <p><del>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch:</del> <a href="https://review.whamcloud.com/47549" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47549</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-15748" title="interop: sanity test_150b: fallocate failed, error Operation not supported, mode 0, offset 62914560, len 4194304" class="issue-link" data-issue-key="LU-15748"><del>LU-15748</del></a> ofd: test fallocate interop for older clients</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: 709a3d698ba73805ae009ea82390960cb3b0ed3e</del></p> <p><del>"Arshad Hussain &lt;arshad.hussain@aeoncomputing.com&gt;"</del> uploaded a new patch: <a href="https://review.whamcloud.com/48274" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48274</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-15748" title="interop: sanity test_150b: fallocate failed, error Operation not supported, mode 0, offset 62914560, len 4194304" class="issue-link" data-issue-key="LU-15748"><del>LU-15748</del></a> tests: fix fallocate interop for older clients</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: b2_14</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: e02fda0198f7b6455dc78a433685504b0c864893</del></p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/47548/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/47548/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15748" title="interop: sanity test_150b: fallocate failed, error Operation not supported, mode 0, offset 62914560, len 4194304" class="issue-link" data-issue-key="LU-15748"><del>LU-15748</del></a> ofd: fix fallocate interop for older clients<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 91cb04ef800a8ae9384777286ee1282a262ee89e</p> <p>Landed for 2.15</p> Related LU-15551 LU-14382 LU-14160 LU-15167 LU-15877 Development Rank 1|i02n4v: Rank (Obsolete) 9223372036854775807 Severity