https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3704 Lustre <p>Sanity-hsm test 21 seems to fail alot. </p> <p>An example:<br/> <a href="https://maloo.whamcloud.com/test_sets/adac0ef6-fb6b-11e2-8c6e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/adac0ef6-fb6b-11e2-8c6e-52540035b04c</a></p> <p>'could not release file' seem to be the slightly more common error. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test_21 Error: 'could not release file' Failure Rate: 32.00% of last 100 executions [all branches] </pre> </div></div> <p>The test logs look like. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity-hsm test 21: Simple release tests == 09:25:28 (1375374328) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.387647 s, 5.4 MB/s Cannot send HSM request (use of /mnt/lustre/d0.sanity-hsm/d21/test_release): Device or resource busy sanity-hsm test_21: @@@@@@ FAIL: could not release file Trace dump: </pre> </div></div> <p>test 3 and 22-24 errors look to all be related. </p> autotest LU-3704

sanity-hsm test_21 - test_24 Error: 'could not release file'

Technical task LU-3647 Major Resolved Cannot Reproduce Bruno Faccini Keith Mannthey HSM Mon, 5 Aug 2013 22:07:06 +0000 Mon, 21 Oct 2013 20:32:51 +0000 Mon, 2 Sep 2013 16:41:00 +0000 Lustre 2.5.0 0 8 <p>I think that test #3 and tests #21-24 are not related.</p> <ul> <li>test #3 is not doing any "hsm_release" commands.<br/> It is failing to copy /etc/passwd into lustre mount point. May be there is a side effect on the system with this file. May be we should replace this copy by another to create the file.</li> </ul> <ul> <li>tests #21-24.<br/> I've looked at test #21. It is failing somewhere at the end of the release process, when closing the file. I'm lacking debug information to detect exactly where this is failing. EBUSY is replied when MDS_CLOSE is replied but OBD_FL_RELEASE flag was not set for some reasons. It could be nice to have more debug for that.</li> </ul> <p>The reason I mention they might be linked is they seem to fail together alot. </p> <p>Another example:<br/> <a href="https://maloo.whamcloud.com/test_sets/0c378cde-ff90-11e2-a3fb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0c378cde-ff90-11e2-a3fb-52540035b04c</a></p> <p>I will open LUs to track the other the other subtests. </p> <p>Another set of test 3, 21-24 failures at: <a href="https://maloo.whamcloud.com/test_sets/aa3d4616-001d-11e3-a856-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/aa3d4616-001d-11e3-a856-52540035b04c</a></p> <p>I looked at the failure of test_3.</p> <p>The failure occurred when the client was trying to set system.posix_acl_access since test_3 uses `cp -p' to copy file. Before doing that, it tried to update the client xattr cache so the calling sequence is as follows:</p> <p>ll_setxattr -&gt; ll_xattr_cache_update -&gt; ll_xattr_cache_refill -&gt; IT_GETXATTR -&gt; mdt_getxattr.</p> <p>However, in mdt_getxattr(), the code snippet below:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> eadatasize = mo_xattr_list(info-&gt;mti_env, next, buf); <span class="code-keyword">if</span> (eadatasize &lt; 0) GOTO(out, rc = eadatasize); v = req_capsule_server_get(info-&gt;mti_pill, &amp;RMF_EAVALS); sizes = req_capsule_server_get(info-&gt;mti_pill, &amp;RMF_EAVALS_LENS); <span class="code-comment">/* Fill out EAVALS and EAVALS_LENS */</span> <span class="code-keyword">for</span> (b = buf-&gt;lb_buf; b &lt; (<span class="code-object">char</span> *)buf-&gt;lb_buf + eadatasize; b += strlen(b) + 1, v += rc) { buf2.lb_buf = v; rc = mdt_getxattr_one(info, b, next, &amp;buf2, med, uc); <span class="code-keyword">if</span> (rc &lt; 0) GOTO(out, rc); sizes[eavallens] = rc; buf2.lb_len -= rc; eavallens++; eavallen += rc; } </pre> </div></div> <p>returned ENODATA from mdt_getxattr_one(). I don't know what's the xattr name due to lack of log on the MDT side.</p> <p>test_3 failures are due to, as Jinshan detailled, ENODATA return during <span class="error">&#91;cp -p /etc/passwd $TESTFILE.append || error &quot;could not create file&quot;&#93;</span> command when trying to set "system.posix_acl_access" XATTR (but why ?). And it is now specifically addressed by <a href="https://jira.whamcloud.com/browse/LU-3730" title="sanity-hsm test_3 Error: &#39;could not create file&#39; " class="issue-link" data-issue-key="LU-3730"><del>LU-3730</del></a> !!</p> <p>test_21-24 failures, as detailled by Aurelien and to be addressed here, looks more like a race/timing issue (like test_21 "wrong block number" errors for <a href="https://jira.whamcloud.com/browse/LU-3700" title="sanity-hsm test_21 Error: &#39;wrong block number&#39; " class="issue-link" data-issue-key="LU-3700"><del>LU-3700</del></a>) due to current usage of "lfs hsm_set --archived --exist &lt;file&gt;" command to mimic "lfs hsm_archive &lt;file&gt;". "wait_request_state $fid ARCHIVE SUCCEED" may need to be also used here ?</p> <p>Will try to setup a platform to reproduce problem, with HSM debug traces enabled on Client/MDS VMs, and running sanity-hsm tests in a loop.</p> <p>I am not able to reproduce problem with current master, even by running sanity-hsm/test_<span class="error">&#91;21-24&#93;</span> in a loop. I only had to avoid/ignore test_24 "atime should be ..." errors since it is still being worked by John in <a href="https://jira.whamcloud.com/browse/LU-3814" title="Fix sanity-hsm test 24: &quot;Release does not change a/mtime (i/o)&quot;" class="issue-link" data-issue-key="LU-3814"><del>LU-3814</del></a>/<a href="https://jira.whamcloud.com/browse/LU-3832" title="osc_io_read_start() ignores O_NOATIME, updates i_atime" class="issue-link" data-issue-key="LU-3832"><del>LU-3832</del></a> I think, when test_24 is actually not run by default.</p> <p>BTW, according to Maloo reports test_<span class="error">&#91;21-24&#93;</span> failures for 'could not release file' stopped around Aug. 14th. And this seems to match with landing of patch for <a href="https://jira.whamcloud.com/browse/LU-3561" title="Add a sanity test for HSM" class="issue-link" data-issue-key="LU-3561"><del>LU-3561</del></a> that brings "real" HSM features (copytool, lfs hsm-commands usage instead of hsm-flags setting) in tests and according tools testing.</p> <p>So my strong assumption is that that this ticket can be closed because unrelated now.</p> <p>To be re-opened in case of re-occurence.</p> Related LU-3730 LU-3700 Development Rank 1|hzvx7z: Rank (Obsolete) 9553