https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9260 Lustre <p>Maloo link: <a href="https://testing.hpdd.intel.com/test_sets/461c1d7e-12fc-11e7-b742-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/461c1d7e-12fc-11e7-b742-5254006e85c2</a></p> <p>test log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>FAILURE SUMMARY: POSIX failures: 6 Test Name Baseline Lustre Report access.43 Succeeded Unresolved chmod.18 Succeeded Unresolved chown.18 Succeeded Unresolved creat.28 Succeeded Unresolved creat.30 Succeeded Unresolved link.23 Succeeded Unresolved FAILURE DESCRIPTIONS: #################################################### Test Name: access.43 Unresolved Test Description: If the implementation supports a read-only file system, EROFS in errno and a return value of -1 on a call to access(path, amode) when write access is requested for a file on a read-only file system. Posix Ref: Component ACCESS Assertion 5.6.3.4-48(C) Test Information: deletion reason: mnt_ro(/dev/loop0, access-d.43) failed </pre> </div></div> client and server: EL7 LU-9260

posix failure: access.43 Unresolved

Bug Major Closed Won't Fix Sarah Liu Sarah Liu Mon, 27 Mar 2017 17:03:23 +0000 Mon, 18 May 2020 17:08:54 +0000 Mon, 18 May 2020 17:08:54 +0000 Lustre 2.10.0 Lustre 2.11.0 Lustre 2.12.0 Lustre 2.10.4 Lustre 2.13.0 Lustre 2.10.6 Lustre 2.12.1 Lustre 2.12.4 0 9 <p>It isn't clear why the test script is trying to mount <tt>/dev/loop0</tt> (which might be the MDT0000 device?) as read-only, instead of remounting the client filesystem read-only? Definitely the right test here would be remounting the client filesystem read-only.</p> <p>It would be worthwhile to look at some older POSIX test logs to see why this is failing now, when (presumably) it didn't fail before.</p> <p>After investigation, it looks like we always use loop devices in posix read-only tests(against lustre) even in EL6, which is obviously wrong, instead it should provide MGS/MDS here. </p> <p>in test access.43, it uses setuprofs to setup the read-only fs.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test43() { char *errptr; int pathok = 0; /* write access on read only file system */ DBUG_ENTER("test43"); testfail = 0; globok = 0; if (setuprofs(t43_dir, t43_file, 'f', (mode_t) MODEANY) != 0) { DBUG_VOID_RETURN; } </pre> </div></div> <p>setuprofs.c</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>if ((rofs = tet_getvar(VSX_ROFS)) == NULL || *rofs == '\0') { xx_rpt(DELETION); in_rpt("deletion reason: parameter %s is not set", VSX_ROFS); DBUG_PRINTF("return", "setuprofs = 1"); DBUG_RETURN(1); } </pre> </div></div> <p>in scripts/vsx-pcts/parameterisations.sh</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>echo "VSX_ROFS=\"$NOSPC_DEV\"" &gt;&gt; $1 </pre> </div></div> <p>in test_sets/SRC/vsxparams</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>NOSPC_DEV="/dev/loop0" </pre> </div></div> <p>The reason why El6 passed before, but EL7 failed, seems because in EL7 loop device can not be cleanup, then remount the same device failed.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>In order to re run the test suites at a later date run the rerun_tests program in vsx0's home directory as the vsx0 user /usr/src/posix/ext4 rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/ioprim/write/d.write/write-d.25': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/ioprim/write/d.write/write-d.16': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/files/mkdir/d.mkdir/mkdir-d.19': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/files/mkfifo/d.mkfifo/mkfifo-d.17': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/files/link/d.link/link-d.25': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/files/rmdir/d.rmdir/rmdir-d.9': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/files/open/d.open/open-d.46': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/files/rename/d.rename/rename-d.17': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/files/unlink/d.unlink/unlink-d.9': Device or resource busy Install and build POSIX test suite successfully! Run POSIX test against lustre filesystem </pre> </div></div> <p>I don't think it is the MDS that should be mounted read-only, but rather the client. The test is running on the client. Also, I don't think that he MDS can be mounted read-only and still work. </p> <p>Yes, I meant to mount lustre client as readonly, not MDS. But for mounting client, it needs to provide MGS info. Before posix read-only tests, it needs to remount FS as read-only, but it always uses /dev/loop0, see above rofs is always /dev/loop0, so it is not testing lustre at all. I think we need replace this /dev/loop0 as mgs:/fsname. </p> <p>I think I understand why we are using the loop device when mounting a readonly file system. In the configuration file TESTROOT/tetexec.cfg, we can set what file system to mount as read only</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># File system which can be mounted read only. # Can be the same as VSX_MOUNT_DEV and VSX_NOSPC_DEV. # Set to <span class="code-quote">"unsup"</span> <span class="code-keyword">if</span> read only file systems are not supported. VSX_ROFS=/dev/loop0 </pre> </div></div> <p>I need to look into how to change this configuration file in our POSIX configuration/setup.</p> <p>a quick update of the processing of the issue. I passed the MGSNID:/FSNAME into the suite to replace the "/dev/loop0" and got following error. The error changes and it seems it try to mount r/w but the test seems test mount read-only. Will add some debug info to those tests and see what happens there.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>FAILURE SUMMARY: POSIX failures: 5 Test Name Baseline Lustre Report chmod.18 Succeeded Unresolved chown.18 Succeeded Unresolved creat.28 Succeeded Unresolved creat.30 Succeeded Unresolved link.23 Succeeded Unresolved FAILURE DESCRIPTIONS: #################################################### Test Name: chmod.18 Unresolved Test Description: If the implementation supports a read-only file system, EROFS in errno and a return value of -1 on a call to chmod(path, mode) when the named file resides on a read-only file system. No change to the file mode shall occur. Posix Ref: Component CHMOD Assertion 5.6.4.4-39(C) Test Information: Test Agency: Unknown System Tested: Unknown Test Date: May 19, 2017 Page 29 of 69 X/OPEN Verification Suite Test-Set Summary Test-Set Summary deletion reason: mnt_rw(onyx-69@tcp:/lustre, chmod-d.18) failed #################################################### </pre> </div></div> <p>Did more investigation and here is what I found<br/> 1. the above changing of using MGS:/FSNAME instead of loop is needed<br/> 2. besides 1, it also needs to change FSTYPE in LSB.tools/userintf.c to lustre, then recompile the code again before running posix against lustre.</p> <p>In that c file, it defines the fstype as ext2</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>/ modify the following for different mountable filesystem types / #define __USERINTF_FSTYPE "ext2" </pre> </div></div> <p>So the process would be<br/> 1. compile posix code and running with default file system<br/> 2. change the c file and recompile<br/> 3. run test with lustre</p> <p>Wei Liu (wei3.liu@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28661" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28661</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9260" title="posix failure: access.43 Unresolved" class="issue-link" data-issue-key="LU-9260"><del>LU-9260</del></a> test: Use the correct mount device when test against lustre<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ee7d8850801800d398e67d0ddd1e041b0e87dc4d</p> <p>The above patch fixes the problem from lustre side, corresponding changes on Posix test suite are also needed which will be made to toolkit. </p> <p>Wei Liu (wei3.liu@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28669" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28669</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9260" title="posix failure: access.43 Unresolved" class="issue-link" data-issue-key="LU-9260"><del>LU-9260</del></a> test: Use correct parameters when test with lustre<br/> Project: build/toolkit<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8a0db0e2de2a1915b1bac6f3180e8bb9bf3c0c0d</p> <p>Minh Diep (minh.diep@intel.com) merged in patch <a href="https://review.whamcloud.com/28669/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28669/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9260" title="posix failure: access.43 Unresolved" class="issue-link" data-issue-key="LU-9260"><del>LU-9260</del></a> test: Use correct parameters when test with lustre<br/> Project: build/toolkit<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4ed7df5ea427b48f43d153c263bac5a41f57307c</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28661/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28661/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9260" title="posix failure: access.43 Unresolved" class="issue-link" data-issue-key="LU-9260"><del>LU-9260</del></a> test: Use the correct mount device when test against lustre<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7b59ed3ab3c9bfae95b9904982869d31a7e65770</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29628" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29628</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9260" title="posix failure: access.43 Unresolved" class="issue-link" data-issue-key="LU-9260"><del>LU-9260</del></a> test: Use the correct mount device when test against lustre<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 9e5e025b0bfc1c4280b2fcd4caabd7fc4db7ea67</p> <p>Seen again with 2.10.54:</p> <p><a href="https://testing.hpdd.intel.com/test_sessions/77fc6fa0-eddf-4fdb-b9ad-8724d84acb75" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/77fc6fa0-eddf-4fdb-b9ad-8724d84acb75</a></p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/29628/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29628/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9260" title="posix failure: access.43 Unresolved" class="issue-link" data-issue-key="LU-9260"><del>LU-9260</del></a> test: Use the correct mount device when test against lustre<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 384cdeac7f40873220193b37fb083970b834fc03</p> <p>The failure occurred at least 9 times in the last week.</p> <p>Here is a failure instance on 2.10.57:<br/> <a href="https://testing.hpdd.intel.com/test_sets/366580ca-fda5-11e7-a7cd-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/366580ca-fda5-11e7-a7cd-52540065bddc</a></p> <p>I searched Maloo for the last week and saw 4 failures in all master branch and interop between b2_10 and master. The failures seen in interop testing between master and b2_9 and/or prior should not count since the client doesn't have the fix. </p> <p>Among the 4 failures, 3 of them were regular configs and 1 was interop config. All of those 3 regular configs failed on ONYX. I suspect this is an env related issue.</p> <p>Cannot reproduce the problem on onyx with physical nodes or vms, will try to provision the node with snapshot and see what I can find</p> <p>Cannot reproduce the problem on vm either, nodes were installed via snapshot</p> <p><a href="https://testing.hpdd.intel.com/test_sessions/c64e9304-0aa5-11e8-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/c64e9304-0aa5-11e8-bd00-52540065bddc</a></p> <p>Should we resolve this issue as fixed in 2.11.0 if it is no longer seen nor reproducible?</p> <p>We are still seeing this issue or something very close to it. Here is an example <a href="https://testing.hpdd.intel.com/test_sets/5fbcc25c-2a4c-11e8-b6a0-52540065bddc." class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5fbcc25c-2a4c-11e8-b6a0-52540065bddc.</a> </p> <p> </p> <p>+1 on master branch:<br/> <a href="https://testing.whamcloud.com/test_sets/8484a974-fdf0-11e8-b837-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8484a974-fdf0-11e8-b837-52540065bddc</a></p> <p>We will not fix this issue because we’ve replaced the POSIX test suite with pjdfstest.</p> Duplicate LU-9261 LU-9262 LU-9264 LU-9732 Development Rank 1|hzz8d3: Rank (Obsolete) 9223372036854775807 Severity