https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10566 Lustre <p>parallel-scale-nfsv4 test_metabench - metabench failed! 1<br/> ^^^^^^^^^^^^^ DO NOT REMOVE LINE ABOVE ^^^^^^^^^^^^^</p> <p>This issue was created by maloo for sarah_lw &lt;wei3.liu@intel.com&gt;</p> <p>This issue relates to the following test suite run: <br/> <a href="https://testing.hpdd.intel.com/test_sets/6e890fe6-fd53-11e7-a6ad-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6e890fe6-fd53-11e7-a6ad-52540065bddc</a></p> <p>test_metabench failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>metabench failed! 1 </pre> </div></div> <p>server: 2.10.57 RHEL7 ldiskfs<br/> client: SLES12SP3</p> <p>test log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== parallel-scale-nfsv4 test metabench: metabench ==================================================== 11:23:59 (1516389839) OPTIONS: METABENCH=/usr/bin/metabench clients=onyx-33vm1,onyx-33vm2 mbench_NFILES=10000 mbench_THREADS=4 onyx-33vm1 onyx-33vm2 mkdir: cannot create directory ‘/mnt/lustre/d0.parallel-scale-nfs’: Read-only file system chmod: cannot access '/mnt/lustre/d0.parallel-scale-nfs/d0.metabench': No such file or directory + /usr/bin/metabench -w /mnt/lustre/d0.parallel-scale-nfs/d0.metabench -c 10000 -C -S + chmod 0777 /mnt/lustre chmod: changing permissions of '/mnt/lustre': Read-only file system dr-xr-xr-x 23 root root 4096 Jan 19 00:29 /mnt/lustre + su mpiuser sh -c "/usr/lib64/mpi/gcc/openmpi/bin/mpirun --mca btl tcp,self --mca btl_tcp_if_include eth0 -mca boot ssh -machinefile /tmp/parallel-scale-nfs.machines -np 8 /usr/bin/metabench -w /mnt/lustre/d0.parallel-scale-nfs/d0.metabench -c 10000 -C -S " [onyx-33vm2:14600] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_ofi: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) [onyx-33vm2:14600] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_psm: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) [onyx-33vm2:14601] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_ofi: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) [onyx-33vm2:14601] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_psm: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) [onyx-33vm1:09898] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_ofi: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) [onyx-33vm1:09898] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_psm: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) [onyx-33vm1:09900] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_ofi: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) [onyx-33vm1:09900] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_psm: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) [onyx-33vm1:09899] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_ofi: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) [onyx-33vm1:09899] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_psm: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) [onyx-33vm2:14602] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_ofi: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) [onyx-33vm2:14602] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_psm: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) [onyx-33vm2:14604] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_ofi: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) [onyx-33vm2:14604] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_psm: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) [onyx-33vm1:09902] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_ofi: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) [onyx-33vm1:09902] mca: base: component_find: unable to open /usr/lib64/mpi/gcc/openmpi/lib64/openmpi/mca_mtl_psm: libpsm_infinipath.so.1: cannot open shared object file: No such file or directory (ignored) Metadata Test &lt;no-name&gt; on 01/19/2018 at 11:23:59 Rank 0 process on node onyx-33vm1 Rank 1 process on node onyx-33vm1 Rank 2 process on node onyx-33vm1 Rank 3 process on node onyx-33vm1 Rank 4 process on node onyx-33vm2 Rank 5 process on node onyx-33vm2 Rank 6 process on node onyx-33vm2 Rank 7 process on node onyx-33vm2 [01/19/2018 11:23:59] FATAL error on process 0 Proc 0: cannot create component d0.parallel-scale-nfs in /mnt/lustre/d0.parallel-scale-nfs/d0.metabench: Read-only file system [onyx-33vm1][[7407,1],1][btl_tcp_frag.c:238:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104) </pre> </div></div> LU-10566

parallel-scale-nfsv4 test_metabench: mkdir: cannot create directory on Read-only file system

Bug Minor Reopened Unresolved WC Triage Sarah Liu Thu, 25 Jan 2018 18:12:01 +0000 Wed, 14 Apr 2021 16:14:13 +0000 Lustre 2.11.0 Lustre 2.10.4 Lustre 2.12.5 Lustre 2.12.6 Lustre 2.12.0 Lustre 2.10.4 0 5 <p>From the top of the suite_log, we see that the file system is 99% full:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1165900 85376 977328 8% /mnt/lustre[MDT:0] lustre-OST0000_UUID 1933276 1801184 10852 99% /mnt/lustre[OST:0] lustre-OST0001_UUID 1933276 1795316 16720 99% /mnt/lustre[OST:1] lustre-OST0002_UUID 1933276 1795232 16776 99% /mnt/lustre[OST:2] lustre-OST0003_UUID 1933276 1795292 16660 99% /mnt/lustre[OST:3] lustre-OST0004_UUID 1933276 1795180 16828 99% /mnt/lustre[OST:4] lustre-OST0005_UUID 1933276 1795300 16596 99% /mnt/lustre[OST:5] lustre-OST0006_UUID 1933276 1803440 8596 100% /mnt/lustre[OST:6] filesystem_summary: 13532932 12580944 103028 99% /mnt/lustre </pre> </div></div> <p>Does this explain why the file system changes to 'read-only' ? I suspect that the NFS file system is read-only, but we should confirm that the Lustre file system is not read-only.</p> <p>We are seeing this issue on failover test sessions with DNE configured and ZFS servers with servers and clients el7:<br/> <a href="https://testing.hpdd.intel.com/test_sets/ed44110c-fd83-11e7-a7cd-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ed44110c-fd83-11e7-a7cd-52540065bddc</a></p> <p>+1 on b2_10<br/> <a href="https://testing.hpdd.intel.com/test_sets/ba76657c-0b5c-11e8-a6ad-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ba76657c-0b5c-11e8-a6ad-52540065bddc</a></p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31231" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31231</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10566" title="parallel-scale-nfsv4 test_metabench: mkdir: cannot create directory on Read-only file system" class="issue-link" data-issue-key="LU-10566">LU-10566</a> test: don't direct lfs df to /dev/null<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b9bc92bd01d9a769ddb4d8669b27fe6db8e7cf54</p> <p>I found that obdfilter-survey test_1c did not clean up propertly</p> <p>=============&gt; Destroy 1 on 10.9.6.12:lustre-OST0000_ecc<br/> error: destroy: invalid objid '3'<br/> destroy OST object &lt;objid&gt; [num <span class="error">&#91;verbose&#93;</span>]<br/> usage: destroy &lt;num&gt; objects, starting at objid &lt;objid&gt;<br/> run &lt;command&gt; after connecting to device &lt;devno&gt;<br/> --device &lt;devno&gt; &lt;command <span class="error">&#91;args ...&#93;</span>&gt;</p> <p>It looks like we are hitting this again with 2.10.59 RHEL 7 ldiskfs servers and RHEL 7 clients.</p> <p>This time, the file system is not almost full. From the suite_log, before we run any parallel-scale-nfsv4 tests,</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>UUID                   1K-blocks        Used   Available Use% Mounted on lustre-MDT0000_UUID      1165900       17368     1045336   2% /mnt/lustre[MDT:0] lustre-OST0000_UUID      1933276       26956     1781868   1% /mnt/lustre[OST:0] lustre-OST0001_UUID      1933276       26944     1785064   1% /mnt/lustre[OST:1] lustre-OST0002_UUID      1933276       31044     1780088   2% /mnt/lustre[OST:2] lustre-OST0003_UUID      1933276       26956     1784908   1% /mnt/lustre[OST:3] lustre-OST0004_UUID      1933276       26948     1784972   1% /mnt/lustre[OST:4] lustre-OST0005_UUID      1933276       26988     1784812   1% /mnt/lustre[OST:5] lustre-OST0006_UUID      1933276       26960     1784960   1% /mnt/lustre[OST:6] filesystem_summary:     13532932      192796    12486672   2% /mnt/lustre  </pre> </div></div> <p>We see the same output from metabench as in the description</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== parallel-scale-nfsv4 test metabench: metabench ==================================================== 08:43:41 (1521017021) OPTIONS: METABENCH=/usr/bin/metabench clients=onyx-30vm5.onyx.hpdd.intel.com,onyx-30vm6 mbench_NFILES=10000 mbench_THREADS=4 onyx-30vm5.onyx.hpdd.intel.com onyx-30vm6 mkdir: cannot create directory '/mnt/lustre/d0.parallel-scale-nfs': Read-only file system chmod: cannot access '/mnt/lustre/d0.parallel-scale-nfs/d0.metabench': No such file or directory + /usr/bin/metabench -w /mnt/lustre/d0.parallel-scale-nfs/d0.metabench -c 10000 -C -S + chmod 0777 /mnt/lustre chmod: changing permissions of '/mnt/lustre': Read-only file system dr-xr-xr-x 23 root root 4096 Mar 14 07:59 /mnt/lustre  </pre> </div></div> <p><a href="https://testing.hpdd.intel.com/test_sets/734ea438-2773-11e8-9e0e-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/734ea438-2773-11e8-9e0e-52540065bddc</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/3b702578-2769-11e8-9e0e-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3b702578-2769-11e8-9e0e-52540065bddc</a></p> <p> </p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31679" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31679</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10566" title="parallel-scale-nfsv4 test_metabench: mkdir: cannot create directory on Read-only file system" class="issue-link" data-issue-key="LU-10566">LU-10566</a> test: debug<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7550568625d2732afeb36a52df36db0109bda82d</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31679/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31679/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10566" title="parallel-scale-nfsv4 test_metabench: mkdir: cannot create directory on Read-only file system" class="issue-link" data-issue-key="LU-10566">LU-10566</a> test: fix nfs exports clean up<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2cdc1ad8b86d013fdb8ffc70ee567284537eee47</p> <p><font color="#000000">Landed for 2.12</font></p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31953" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31953</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10566" title="parallel-scale-nfsv4 test_metabench: mkdir: cannot create directory on Read-only file system" class="issue-link" data-issue-key="LU-10566">LU-10566</a> test: fix nfs exports clean up<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 5a86bc8daac771e428ad839f27ea1542a4a40f48</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/31953/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31953/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10566" title="parallel-scale-nfsv4 test_metabench: mkdir: cannot create directory on Read-only file system" class="issue-link" data-issue-key="LU-10566">LU-10566</a> test: fix nfs exports clean up<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 741347aafb8053d02294650add007e1bf050e978</p> <p>hit this again on b2_10 2.10.6-rc2 zfs DNE</p> <p><a href="https://testing.whamcloud.com/test_sets/05a4f148-ef60-11e8-bfe1-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/05a4f148-ef60-11e8-bfe1-52540065bddc</a></p> <p>I'm reopening this ticket because we are still seeing the read-only file system problem for 2.12.5 RC1 at <a href="https://testing.whamcloud.com/test_sets/bd833e99-d557-4ec6-a768-91440b98b55e" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/bd833e99-d557-4ec6-a768-91440b98b55e</a> .</p> <p>Maybe <a href="https://jira.whamcloud.com/browse/LU-12231" title="parallel-scale-nfsv4 test racer_on_nfs fails with &#39;test_racer_on_nfs failed with 1&#39;" class="issue-link" data-issue-key="LU-12231"><del>LU-12231</del></a> is the same issue and this one can be closed since <a href="https://jira.whamcloud.com/browse/LU-12231" title="parallel-scale-nfsv4 test racer_on_nfs fails with &#39;test_racer_on_nfs failed with 1&#39;" class="issue-link" data-issue-key="LU-12231"><del>LU-12231</del></a> is still open?</p> Related LU-10663 LU-10851 LU-9892 LU-12231 Development Rank 1|hzzrqn: Rank (Obsolete) 9223372036854775807 Severity