https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4428 Lustre <p>performance-sanity test_5 failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Total disk size: 16257024 block-softlimit: 16258048 block-hardlimit: 17070950 inode-softlimit: 55785 inode-hardlimit: 58574 &lt;~snip~&gt; ===== mdsrate-lookup-1dir.sh Test preparation: creating 63715 files. + /usr/lib64/lustre/tests/mdsrate --mknod --dir /mnt/lustre/mdsrate/lookup --nfiles 63715 --filefmt 'f%%d' + chmod 0777 /mnt/lustre drwxrwxrwx 9 root root 99840 Apr 14 2107 /mnt/lustre + su mpiuser sh -c "/usr/lib64/openmpi/bin/mpirun -mca boot ssh -machinefile /tmp/mdsrate-lookup-1dir.machines -np 2 /usr/lib64/lustre/tests/mdsrate --mknod --dir /mnt/lustre/mdsrate/lookup --nfiles 63715 --filefmt 'f%%d' " &lt;~snip~&gt; rank 1: mknod(f59999) error: Disk quota exceeded rank 0: mknod(f58728) error: Disk quota exceeded </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/fc281616-73c4-11e3-b4ff-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fc281616-73c4-11e3-b4ff-52540035b04c</a></p> <br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/5/">http://build.whamcloud.com/job/lustre-b2_5/5/</a><br/> FSTYPE=zfs<br/> <br/> OSTCOUNT=2<br/> OSTSIZE=8388608<br/> LU-4428

performance-sanity test_5: mknod(f59999) error: Disk quota exceeded

Bug Major Open Unresolved Nathaniel Clark Jian Yu zfs Fri, 3 Jan 2014 12:27:47 +0000 Thu, 11 Apr 2019 16:24:10 +0000 Lustre 2.5.1 Lustre 2.4.3 Lustre 2.5.3 Lustre 2.10.2 Lustre 2.10.4 Lustre 2.10.5 Lustre 2.10.7 Lustre 2.12.1 0 6 <p>More instance on Lustre b2_5 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/6355c6e8-7a15-11e3-bce8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6355c6e8-7a15-11e3-bce8-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/249915d8-7ec3-11e3-a0a8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/249915d8-7ec3-11e3-a0a8-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/8728f446-9ac7-11e3-96fb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8728f446-9ac7-11e3-96fb-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/39/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/39/</a> (2.5.1 RC1)<br/> Distro/Arch: RHEL6.5/x86_64<br/> FSTYPE=zfs</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/7a0a591c-a607-11e3-8a1b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7a0a591c-a607-11e3-8a1b-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/73/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/73/</a> (2.4.3 RC1)<br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p><a href="https://maloo.whamcloud.com/test_sets/e70e3c7c-ac60-11e3-81d7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e70e3c7c-ac60-11e3-81d7-52540035b04c</a></p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_5/47/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/47/</a><br/> FSTYPE=zfs</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/99e3e576-cb70-11e3-95c9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/99e3e576-cb70-11e3-95c9-52540035b04c</a></p> <p>Nathaniel</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/61/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/61/</a><br/> FSTYPE=zfs</p> <p>The same failure occurred: <a href="https://maloo.whamcloud.com/test_sets/50a6f7f0-ebe5-11e3-82b2-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/50a6f7f0-ebe5-11e3-82b2-52540035b04c</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">inode-softlimit: 55785 inode-hardlimit: 58574 </pre> </div></div> <p>The hardlimit was set to 58574.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">/usr/lib64/lustre/tests/mdsrate --mknod --dir /mnt/lustre/mdsrate/lookup --nfiles 63715 --filefmt <span class="code-quote">'f%%d'</span> </pre> </div></div> <p>However, mdsrate-lookup-1dir.sh want to create 63715 files.</p> <p>test-framework.sh set hardlimit based on total inodes for whole cluster. (see setup_quota() ), but mdsrate-lookup-1dir.sh creates files based on free inodes on MDT. (see inodes_available()). I think they'd be unified.</p> <p>This may have something to do with how free inodes are calculated, I believe it is the minimum of what's available across OSTs and what's available on the MDT. So if that's the case, this would be an issue with the test.</p> <p>Lustre Build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/86/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_5/86/</a> (2.5.3 RC1)<br/> FSTYPE=zfs</p> <p>The same failure occurred: <a href="https://testing.hpdd.intel.com/test_sets/48d589c2-311f-11e4-b503-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/48d589c2-311f-11e4-b503-5254006e85c2</a></p> <p>Testing on b2_10 (after 2.10.2) (302e4ec4bd923b79690d8c9e7004e0ae0c67be98)<br/> <a href="https://testing.hpdd.intel.com/test_sets/e98a2540-e60b-11e7-8027-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e98a2540-e60b-11e7-8027-52540065bddc</a></p> <p>MDS dmesg:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[29392.140663] Lustre: lustre-MDT0000: Connection restored to 10.9.6.120@tcp (at 10.9.6.120@tcp) [29392.144060] Lustre: Skipped 1 previous similar message [29413.881762] LustreError: 11-0: lustre-OST0005-osc-MDT0000: operation ost_connect to node 10.9.6.120@tcp failed: rc = -11 [29413.885108] LustreError: Skipped 1 previous similar message </pre> </div></div> <p>+1 on b2_10 <a href="https://testing.whamcloud.com/test_sets/be7048b2-5c17-11e8-b303-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/be7048b2-5c17-11e8-b303-52540065bddc</a></p> Duplicate LU-5146 Development Rank 1|hzwc4v: Rank (Obsolete) 12168 Severity