https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3522 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/903643c8-ddca-11e2-85a3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/903643c8-ddca-11e2-85a3-52540035b04c</a>.</p> <p>The sub-test test_iozone failed with the following error:</p> <blockquote> <p>iozone (2) failed</p></blockquote> <p>test log shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Error writing block 7269, fd= 3 write: No space left on device iozone: interrupted exiting iozone sanity-benchmark test_iozone: @@@@@@ FAIL: iozone (2) failed </pre> </div></div> server and client : lustre-master build# 1536 zfs LU-3522

sanity-benchmark test_iozone: "no space left on device" on ZFS

Bug Critical Resolved Fixed Nathaniel Clark Maloo Thu, 27 Jun 2013 21:28:05 +0000 Mon, 9 Oct 2017 19:55:23 +0000 Mon, 9 Oct 2017 19:55:23 +0000 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.5.0 Lustre 2.8.0 Lustre 2.10.0 0 8 <p>Either it both the test script should be fixed to check the available space before starting, and stripe the file over all OSTs or create multiple output files. </p> <p>Lustre Branch: b2_4<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/27/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/27/</a><br/> Distro/Arch: RHEL6.4/x86_64</p> <p>The failure occurred regularly on Lustre b2_4 branch with ZFS backend filesystem:<br/> <a href="https://maloo.whamcloud.com/test_sets/c39c4b36-fd73-11e2-9fdb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c39c4b36-fd73-11e2-9fdb-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/9a3acd76-fd0a-11e2-9fdb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9a3acd76-fd0a-11e2-9fdb-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/6cb6014e-ed89-11e2-8e3a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6cb6014e-ed89-11e2-8e3a-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/191be2e6-ce18-11e2-96ef-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/191be2e6-ce18-11e2-96ef-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/f29840d6-cb67-11e2-a1fe-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f29840d6-cb67-11e2-a1fe-52540035b04c</a></p> <p>Nathaniel</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>It looks like the issue is the O_DIRECT nature of the iozone run.</p> <p>SLES11 SP2 client also hit this issue with ldiskfs:</p> <p><a href="https://maloo.whamcloud.com/test_sets/4ee7f69a-029c-11e3-b384-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4ee7f69a-029c-11e3-b384-52540035b04c</a></p> <p>It appears that when doing O_DIRECT, the transfer runs out of grant space before completing and thus gets an ENOSPC error back (this is zfs only).</p> <p>The difference between ldiskfs and zfs seems to be in the calculated grant request space, namely ofd_grant_from_cli()'s conversion factor for ldiskfs is 1, whereas for zfs it's 32. This is because the ofd_grant_compat() assumes that if the ofd_blockbits (set from statfs which gives the largest available block size in zfs) is greater than 4KB (12 bits) that it should assume the worst case, and shift out by that amount (which in this case is 5 bits).</p> <p><a href="http://review.whamcloud.com/7402" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7402</a></p> <p>I put a brief description of what needs to be done into <a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a>. I thought that Jinshan and Johann had worked up a more detailed design for this, but I can't find it.</p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/44/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/44/</a> (2.4.1 RC1)<br/> Distro/Arch: RHEL6.4/x86_64 + FC18/x86_64 (Server + Client)<br/> FSTYPE=ldiskfs</p> <p>sanity-benchmark test iozone hit the same failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/becb9218-14ef-11e3-ac48-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/becb9218-14ef-11e3-ac48-52540035b04c</a></p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/44/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/44/</a> (2.4.1 RC1)<br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p>sanity-benchmark test iozone also hit the same failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/e004601a-1556-11e3-8938-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e004601a-1556-11e3-8938-52540035b04c</a></p> <p>lustre-master build #1652 also hit this issue on ldiskfs:</p> <p><a href="https://maloo.whamcloud.com/test_sets/c30260ee-15b4-11e3-8938-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c30260ee-15b4-11e3-8938-52540035b04c</a></p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/45/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/45/</a> (2.4.1 RC2)<br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=ldiskfs</p> <p>sanity-benchmark test iozone also hit the same failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/497a00b4-182b-11e3-b39a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/497a00b4-182b-11e3-b39a-52540035b04c</a></p> <p>The current state of this bug seems to be: Behaving as expected, wait for OBD_CONNECT_GRANT_PARAM to be supported on the client (<a href="https://jira.whamcloud.com/browse/LU-2049" title="add support for OBD_CONNECT_GRANT_PARAM" class="issue-link" data-issue-key="LU-2049"><del>LU-2049</del></a>).</p> <p>Lai</p> <p>Are you able to assist with this one?</p> <p>Peter</p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/47/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/47/</a><br/> Distro/Arch: RHEL6.4/x86_64(server), SLES11SP2/x86_64(client)<br/> FSTYPE=ldiskfs</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/e5731cd8-432f-11e3-8676-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e5731cd8-432f-11e3-8676-52540035b04c</a></p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/47/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/47/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/5d2a22d4-43a9-11e3-942a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/5d2a22d4-43a9-11e3-942a-52540035b04c</a></p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/47/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/47/</a><br/> Distro/Arch: RHEL6.4/x86_64</p> <p>FSTYPE=zfs<br/> MDSCOUNT=1<br/> MDSSIZE=2097152<br/> OSTCOUNT=2<br/> OSTSIZE=2097152</p> <p>sanity-benchmark test iozone failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 3844920 512 Error writing block 7508, fd= 3 iozone: interrupted exiting iozone sanity-benchmark test_iozone: @@@@@@ FAIL: iozone (1) failed </pre> </div></div> <p>Dmesg on client node showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: min OST has 2014336kB available, using 3844920kB file size LustreError: 17272:0:(vvp_io.c:1088:vvp_io_commit_write()) Write page 961062 of inode ffff88007af981f8 failed -28 </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/51ec0c9c-444f-11e3-8472-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/51ec0c9c-444f-11e3-8472-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/58/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/58/</a><br/> Distro/Arch: RHEL6.4/x86_64</p> <p>FSTYPE=zfs<br/> MDSCOUNT=1<br/> MDSSIZE=2097152<br/> OSTCOUNT=2<br/> OSTSIZE=2097152</p> <p>sanity-benchmark test iozone failed again:<br/> <a href="https://maloo.whamcloud.com/test_sets/f6ed7dd6-5604-11e3-8e94-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f6ed7dd6-5604-11e3-8e94-52540035b04c</a></p> <p>More instances on Lustre b2_4 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/bc739fb2-6353-11e3-8c76-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/bc739fb2-6353-11e3-8c76-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/75b2103c-6366-11e3-8ae4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/75b2103c-6366-11e3-8ae4-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/06fb5b2e-627d-11e3-a8fd-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/06fb5b2e-627d-11e3-a8fd-52540035b04c</a></p> <p>I have similar failure in interop testing with master branch ldiskfs</p> <p><a href="https://testing.hpdd.intel.com/test_sets/b4f55f66-250b-11e5-8009-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b4f55f66-250b-11e5-8009-5254006e85c2</a></p> Blocker LU-2049 Duplicate LU-3906 LU-4042 LU-2049 Development Rank 1|hzvu5j: Rank (Obsolete) 8858 Severity