https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6707 Lustre <p><a href="https://testing.hpdd.intel.com/test_sets/209ce6aa-7ee3-11e4-ab67-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/209ce6aa-7ee3-11e4-ab67-5254006e85c2</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>posix test_1: @@@@@@ FAIL: /dev/loop/1 and /dev/loop1 gone? </pre> </div></div> server: lustre-master #2770 RHEL6<br/> client: EL7 LU-6707

EL7 client cannot find loop device for posix test

Bug Critical Resolved Fixed Jian Yu Sarah Liu triage Sat, 31 Jan 2015 00:36:56 +0000 Wed, 9 May 2018 17:05:35 +0000 Thu, 16 Mar 2017 21:59:47 +0000 Lustre 2.9.0 Lustre 2.10.0 Lustre 2.10.0 0 12 <p>Lustre build: <a href="https://build.hpdd.intel.com/job/lustre-b_ieel2_0/176/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b_ieel2_0/176/</a><br/> Distro/Arch: RHEL6.6/x86_64 (server), RHEL7.1/x86_64 (client)</p> <p>The same failure occurred:<br/> <a href="https://testing.hpdd.intel.com/test_sets/80ca46e0-efcb-11e4-bd0b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/80ca46e0-efcb-11e4-bd0b-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/34267458-ee9f-11e4-a2ce-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/34267458-ee9f-11e4-a2ce-5254006e85c2</a></p> <p>On a RHEL 7.1 client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ls /dev/loop* /dev/loop-control </pre> </div></div> <p>There is no loop device.</p> <p>This need to investigate by testing manually. I can't find anything that point to TEI.</p> <p>Sarah,<br/> Would you be able to test this one manually and post the results here?</p> <p>Jodi,</p> <p>Sure, I will do it today and update the ticket when I have some results</p> <p>Right after provision EL7 client, nothing has been run, there is no loop device under /dev, will do more investigation.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@eagle-39vm3 ~]# ls /dev|grep loop loop-control [root@eagle-39vm3 ~]# rpm -a|grep lustre [root@eagle-39vm3 ~]# rpm -qa|grep lustre lustre-client-modules-2.7.52-3.10.0_229.1.2.el7.x86_64_gbd07c02.x86_64 lustre-iokit-2.7.52-3.10.0_229.1.2.el7.x86_64_gbd07c02.x86_64 lustre-client-2.7.52-3.10.0_229.1.2.el7.x86_64_gbd07c02.x86_64 lustre-client-tests-2.7.52-3.10.0_229.1.2.el7.x86_64_gbd07c02.x86_64 [root@eagle-39vm3 ~]# uname -a Linux eagle-39vm3.eagle.hpdd.intel.com 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 27 03:04:26 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux [root@eagle-39vm3 ~]# </pre> </div></div> <p>This is what I got from EL6</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@eagle-39vm1 ~]# uname -a Linux eagle-39vm1.eagle.hpdd.intel.com 2.6.32-504.16.2.el6_lustre.gec772b8.x86_64 #1 SMP Thu Apr 30 21:09:20 PDT 2015 x86_64 x86_64 x86_64 GNU/Linux [root@eagle-39vm1 ~]# ls /dev|grep loop loop0 loop1 loop2 loop3 loop4 loop5 loop6 loop7 </pre> </div></div> <p>On EL7, there is no loop device has been set up during system initialization, cannot find the setup commands in /etc/rc.d/init.d/functions</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@eagle-39vm3 init.d]# pwd /etc/rc.d/init.d [root@eagle-39vm3 init.d]# grep -r "losetup" . [root@eagle-39vm3 init.d]# uname -a Linux eagle-39vm3.eagle.hpdd.intel.com 3.10.0-229.1.2.el7.x86_64 #1 SMP Fri Mar 27 03:04:26 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux [root@eagle-39vm3 init.d]# rpm -qf functions initscripts-9.49.17-1.el7_0.1.x86_64 [root@eagle-39vm3 init.d]# </pre> </div></div> <p>While on EL6, /etc/rc.d/init.d/functions has the following commands.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@eagle-39vm1 init.d]# grep -r "losetup" . ./functions: losetup $dev &gt; /dev/null 2&gt;&amp;1 &amp;&amp; \ ./functions: losetup -d $dev [root@eagle-39vm1 init.d]# rpm -qf functions initscripts-9.03.46-1.el6.centos.1.x86_64 [root@eagle-39vm1 init.d]# pwd /etc/rc.d/init.d [root@eagle-39vm1 init.d]# </pre> </div></div> <p>I think this is the reason why EL7 doesn't have loop devices setup after booting. May be our provision script should handle this.</p> <p>This needs more investigation, but if EL7 doesn't have the loop devices created, perhaps posix testsuite need to do that. I wonder if we are missing any packages in EL7</p> <p>I have tried that the loop device can be created manually with losetup, I used the same provision command for EL6 and EL7</p> <p>The same issue is on SLES12 client:<br/> <a href="https://testing.hpdd.intel.com/test_sets/1bbaa112-f8f1-11e4-94ea-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1bbaa112-f8f1-11e4-94ea-5254006e85c2</a></p> <p>so looks like starting the new kernel, the default loop devices are not there. This means we have to create it from posix test suit(Lustre QE) should be owned by lustre QE group. </p> <p>Yu Jian will update this ticket with what needs to be done (think is the posix test suite that needs to be modified/updated). <br/> Sarah will work with YuJian to get this done.<br/> QE will own taking these types of changes in the future.</p> <p>Hi Minh,<br/> Can we add some codes into the node provisioning procedure to check and create loop devices on the node? Similar like what we did for creating sanityusr, quota_usr, mpiuser, etc.<br/> This will avoid posix test failing with test environment issues while creating loop devices in itself.</p> <p>Hi Yujian</p> <p>I think the testsuite has to take care of creating what's needed to run the test ( and remove it afterward ideally). creating sanityusr... was not the right way to begin with.<br/> Someone might run posix test on non-autotest system and will fail and has to create loop device manually. If we use posix to create then it's all there.</p> <p>OK, so the changes will be adding functions into test-framework.sh to check, create and remove loop devices. And in lustre/tests/posix.sh, make setup_loop_dev() and cleanup_loop_dev() call those functions.</p> <p>it turns out that if loading loop module with options max_loop=8 , the system will have loop devices</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@eagle-54vm1 modprobe.d]# modprobe loop max_loop=8 [ 307.399441] loop: module loaded [root@eagle-54vm1 modprobe.d]# ls /dev/loop* /dev/loop0 /dev/loop2 /dev/loop4 /dev/loop6 /dev/loop-control /dev/loop1 /dev/loop3 /dev/loop5 /dev/loop7 </pre> </div></div> <p>Wei Liu (wei3.liu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15130" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15130</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6707" title="EL7 client cannot find loop device for posix test" class="issue-link" data-issue-key="LU-6707"><del>TEI-3114</del></a> test: load loop module to have loop devices<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1612288b7ca7b5ac5d8585fd3d96d395c3da2859</p> <p>This issue is blocked by TEI-3627, posix test needs stropts.h and xtitypes.h which are missing from current EL7 build.</p> <p>RHEL7 uses /dev/loop-control to manage loop devices. So haven't any /dev/loopXX device be created in advance. But you can stilll use losetup to maintenance loop device. </p> <p>Here is the test report of POSIX compliance testing on RHEL 7.1 client and server:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/86d3c726-4b17-11e5-9143-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/86d3c726-4b17-11e5-9143-5254006e85c2</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>FAILURE SUMMARY: POSIX failures: 6 Test Name Baseline Lustre Report access.43 Succeeded Unresolved chmod.18 Succeeded Unresolved chown.18 Succeeded Unresolved creat.28 Succeeded Unresolved creat.30 Succeeded Unresolved link.23 Succeeded Unresolved </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>FAILURE DESCRIPTIONS: #################################################### Test Name: access.43 Unresolved Test Description: If the implementation supports a read-only file system, EROFS in errno and a return value of -1 on a call to access(path, amode) when write access is requested for a file on a read-only file system. Posix Ref: Component ACCESS Assertion 5.6.3.4-48(C) Test Information: deletion reason: mnt_ro(/dev/loop0, access-d.43) failed #################################################### Test Name: chmod.18 Unresolved Test Description: If the implementation supports a read-only file system, EROFS in errno and a return value of -1 on a call to chmod(path, mode) when the named file resides on a read-only file system. No change to the file mode shall occur. Posix Ref: Component CHMOD Assertion 5.6.4.4-39(C) Test Information: deletion reason: mnt_ro(/dev/loop0, chmod-d.18) failed #################################################### Test Name: chown.18 Unresolved Test Description: If the implementation supports a read-only file system, EROFS in errno and a return value of -1 on a call to chown(path, owner, group) when the named file resides on a read-only file system. No change shall be made to the owner and group of the file. Posix Ref: Component CHOWN Assertion 5.6.5.4-40(C) Test Information: deletion reason: mnt_ro(/dev/loop0, chown-d.18) failed #################################################### Test Name: creat.28 Unresolved Test Description: EROFS in errno and a return value of -1 on a call to creat(path, mode) when: a. the file exists and the named file resides on a read-only file system; b. the named file is to reside on a read-only file system and the file does not exist. The time related elements st_ctime and st_mtime field of the parent directory shall not be updated and the file shall not be truncated. Posix Ref: Component CREAT Assertion 5.3.2.4-55(C) Posix Ref: Component CREAT Assertion 5.3.2.4-56(C) Test Information: deletion reason: mnt_ro(/dev/loop0, ./creat-d.28) failed #################################################### Test Name: creat.30 Unresolved Test Description: ENOSPC in errno and a return value of -1 on a call to creat(path) when the directory or file system which would contain the new file cannot be extended. Posix Ref: Component CREAT Assertion 5.3.2.4-53(B) Test Information: File system not set up correctly for ENOSPC tests #################################################### Test Name: link.23 Unresolved Test Description: EROFS in errno and a return value of -1 on a call to link() when the requested link requires writing in a directory on a read-only file system. Posix Ref: Component LINK Assertion 5.3.4.4-63(C) Test Information: deletion reason: mnt_ro(/dev/loop0, link-d.23) failed #################################################### posix test_1: @@@@@@ FAIL: Run POSIX testsuite on /mnt/lustre failed </pre> </div></div> <p>Except creat.30, other failures are related to /dev/loop0.</p> <p>In addition, the following errors occurred before running POSIX test against Lustre filesystem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/files/mkfifo/d.mkfifo/mkfifo-d.17': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/files/rmdir/d.rmdir/rmdir-d.9': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/files/mkdir/d.mkdir/mkdir-d.19': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/files/link/d.link/link-d.25': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/files/unlink/d.unlink/unlink-d.9': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/files/open/d.open/open-d.46': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/files/rename/d.rename/rename-d.17': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/ioprim/write/d.write/write-d.16': Device or resource busy rm: cannot remove '/usr/src/posix/ext4/TESTROOT/tset/POSIX.os/ioprim/write/d.write/write-d.25': Device or resource busy </pre> </div></div> <p><tt>lsof</tt> and <tt>fuser</tt> output nothing:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lsof +D /usr/src/posix/ext4/TESTROOT/ # lsof +D /usr/src/posix/ext4/TESTROOT/tset/POSIX.os/ioprim/write/d.write/write-d.16/ # fuser -u /usr/src/posix/ext4/TESTROOT/tset/POSIX.os/ioprim/write/d.write/write-d.16/ # ls -la /usr/src/posix/ext4/TESTROOT/tset/POSIX.os/ioprim/write/d.write/write-d.16/ total 2 drwxr-xr-x 2 root root 1024 Aug 25 17:34 . drwxrwsrwx 4 vsx0 vsxg0 1024 Aug 25 17:34 .. </pre> </div></div> <p>Still investigating.</p> <p>master, build# 3264, 2.7.64 tag<br/> Regression:EL7.1 Server/EL7.1 Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/7acb9552-9f37-11e5-ba94-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7acb9552-9f37-11e5-ba94-5254006e85c2</a></p> <p>Server: 2.5.5, b2_5_fe/62<br/> Client: Master, Build# 3266, Tag 2.7.64 , RHEL 7 <br/> <a href="https://testing.hpdd.intel.com/test_sets/27d8f730-a05f-11e5-90cc-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/27d8f730-a05f-11e5-90cc-5254006e85c2</a></p> <p>Another instance for EL7.1 Server/EL7.1 Client - DNE<br/> Master , Build# 3270<br/> <a href="https://testing.hpdd.intel.com/test_sets/74fdd71e-a26d-11e5-bdef-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/74fdd71e-a26d-11e5-bdef-5254006e85c2</a></p> <p>Another instance found for interop : 2.5.5 Server/EL7 Client<br/> Server: 2.5.5, b2_5_fe/62<br/> Client: master, build# 3303, RHEL 7<br/> <a href="https://testing.hpdd.intel.com/test_sets/89223572-bb0a-11e5-87b4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/89223572-bb0a-11e5-87b4-5254006e85c2</a></p> <p>Encountered another instance for tag 2.7.66 for FULL - EL7.1 Server/EL7.1 Client , master , build# 3314.<br/> <a href="https://testing.hpdd.intel.com/test_sets/c3cee9b2-ca88-11e5-84d3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c3cee9b2-ca88-11e5-84d3-5254006e85c2</a></p> <p>Another instance for FULL - EL7.1 Server/EL7.1 Client - DNE, master, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/b64f210c-cac5-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b64f210c-cac5-11e5-9609-5254006e85c2</a></p> <p>Another instance found for interop tag 2.7.66 - 2.7.1 Server/EL7 Client, build# 3316<br/> <a href="https://testing.hpdd.intel.com/test_sets/9f032636-ccde-11e5-8b0e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9f032636-ccde-11e5-8b0e-5254006e85c2</a></p> <p>Another instance found for interop tag 2.7.66 - 2.5.5 Server/EL7 Client, build# 3316<br/> <a href="https://testing.hpdd.intel.com/test_sets/6241fb0c-ccc8-11e5-b80c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6241fb0c-ccc8-11e5-b80c-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 - EL7.1 Server/EL7.1 Client, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/c3cee9b2-ca88-11e5-84d3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c3cee9b2-ca88-11e5-84d3-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 -EL7.1 Server/EL7.1 Client - DNE, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/b64f210c-cac5-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b64f210c-cac5-11e5-9609-5254006e85c2</a></p> <p>Another instance found for interop - EL7 Server/2.7.1 Client, tag 2.7.90. <br/> <a href="https://testing.hpdd.intel.com/test_sessions/495aabae-d306-11e5-be5c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/495aabae-d306-11e5-be5c-5254006e85c2</a><br/> Another instance found for interop - 2.7.1 Server/EL7 Client, tag 2.7.90. <br/> <a href="https://testing.hpdd.intel.com/test_sessions/3b9722f8-d2f8-11e5-bf08-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/3b9722f8-d2f8-11e5-bf08-5254006e85c2</a><br/> Another instance found for interop - 2.5.5 Server/EL7 Client, tag 2.7.90. <br/> <a href="https://testing.hpdd.intel.com/test_sessions/ba9d84fe-d300-11e5-be5c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/ba9d84fe-d300-11e5-be5c-5254006e85c2</a></p> <p>This ticket is only for the loop device issue, header missing problem is tracking under <a href="https://jira.whamcloud.com/browse/LU-9102" title="Header files are missing from EL7 which needed by the posix test suite" class="issue-link" data-issue-key="LU-9102"><del>LU-9102</del></a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/15130/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/15130/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6707" title="EL7 client cannot find loop device for posix test" class="issue-link" data-issue-key="LU-6707"><del>LU-6707</del></a> test: load loop module to have loop devices<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b7b080bcb0e2cac15174de44b1b822fc11feec02</p> <p>Landed for 2.10</p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/27012" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27012</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6707" title="EL7 client cannot find loop device for posix test" class="issue-link" data-issue-key="LU-6707"><del>LU-6707</del></a> tests: Add ability to skip tests in POSIX<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: dea19ff2b5b237e4a5d0df6162f88bbdd8ed4893</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/27012/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27012/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6707" title="EL7 client cannot find loop device for posix test" class="issue-link" data-issue-key="LU-6707"><del>LU-6707</del></a> tests: Add ability to skip tests in POSIX<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 95c0d6fa814657b04375d2276e8c1a4d484d028d</p> <p>2.10.1 b26 &lt;--&gt; 2.9.0 b22:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/86b672ef-d6c1-431c-a6a9-c5b9f5358da1" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/86b672ef-d6c1-431c-a6a9-c5b9f5358da1</a></p> Related LU-9102 LU-7758 Development Rank 1|hzx5cv: Rank (Obsolete) 17309 Severity