https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9826 Lustre <p>This issue was created by maloo for Jinshan Xiong &lt;jinshan.xiong@intel.com&gt;</p> <p>Please provide additional information about the failure here.</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/b60d81b2-770c-11e7-8db2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b60d81b2-770c-11e7-8db2-5254006e85c2</a>.</p> <p>Detail message about the failure</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>unlink(/tmp/t32/mnt/lustre/t32_qf_1) error: No such file or directory total: 1 unlinks in 0 seconds: inf unlinks/second CMD: trevis-5vm12 test -f /tmp/t32/list CMD: trevis-5vm12 test -f /tmp/t32/list2 /tmp/t32/mnt/lustre/striped_dir_old /usr/lib64/lustre/tests CMD: trevis-5vm12 cat /tmp/t32/list2 /usr/lib64/lustre/tests CMD: trevis-5vm12 /usr/sbin/lctl set_param -n osd*.*.force_sync=1 dd: failed to open '/tmp/t32/mnt/lustre/tmp_file': No space left on device conf-sanity test_32b: @@@@@@ FAIL: dd failed Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4968:error_noexit() = /usr/lib64/lustre/tests/conf-sanity.sh:2140:t32_test() = /usr/lib64/lustre/tests/conf-sanity.sh:2349:test_32b() = /usr/lib64/lustre/tests/test-framework.sh:5256:run_one() = /usr/lib64/lustre/tests/test-framework.sh:5295:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:5142:run_test() = /usr/lib64/lustre/tests/conf-sanity.sh:2353:main() </pre> </div></div> LU-9826

conf-sanity: test 32b failed with 1

Bug Minor Resolved Fixed Sarah Liu Maloo Thu, 3 Aug 2017 22:59:32 +0000 Thu, 24 Aug 2017 15:09:19 +0000 Thu, 17 Aug 2017 04:37:07 +0000 Lustre 2.10.1 Lustre 2.11.0 0 7 <p>it failed at:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> sync; sleep 5; sync $r $LCTL set_param -n osd*.*.force_sync=1 dd <span class="code-keyword">if</span>=/dev/zero of=$tmp/mnt/lustre/tmp_file bs=10k count=10 </pre> </div></div> <p>The log said:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00000001:0.0:1501803264.544507:0:21733:0:(osp_dev.c:738:osp_statfs()) Process entered 00000004:00001000:0.0:1501803264.544508:0:21733:0:(osp_dev.c:774:osp_statfs()) t32fs-OST0000-osc-MDT0000: 20128 blocks, 5504 free, 32 avail, 1836 files, 1349 free files 00000004:00000001:0.0:1501803264.544509:0:21733:0:(osp_dev.c:775:osp_statfs()) Process leaving (rc=0 : 0 : 0) 00020000:00000001:0.0:1501803264.544509:0:21733:0:(lod_qos.c:205:lod_statfs_and_check()) Process leaving (rc=18446744073709551588 : -28 : ffffffffffffffe4) 00020000:00000001:0.0:1501803264.544510:0:21733:0:(lod_qos.c:296:lod_qos_statfs_update()) Process leaving 00020000:00000010:0.0:1501803264.544511:0:21733:0:(lod_qos.c:2067:lod_qos_prep_create()) kmalloced 'stripe': 8 at ffff8800a8dec110. 00020000:00001000:0.0:1501803264.544512:0:21733:0:(lod_qos.c:2074:lod_qos_prep_create()) tgt_count 1 stripenr 1 00020000:00000001:0.0:1501803264.544512:0:21733:0:(lod_qos.c:1443:lod_alloc_qos()) Process entered 00020000:00000001:0.0:1501803264.544513:0:21733:0:(lod_qos.c:1464:lod_alloc_qos()) Process leaving via out_nolock (rc=18446744073709551605 : -11 : 0xfffffffffffffff5) </pre> </div></div> <p>And:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00002000:00000024:1.0:1501803265.168716:0:23940:0:(ofd_obd.c:740:ofd_statfs()) blocks cached 0 granted 20447232 pending 0 free 22544384 avail 20447232 00002000:00000020:1.0:1501803265.168717:0:23940:0:(tgt_grant.c:150:tgt_check_export_grants()) t32fs-OST0000: cli 07f7f756-bbb3-3f62-5432-0eeecfceef6b/ffff8801225b6400 dirty 0 pend 0 grant 131072 00002000:00000020:1.0:1501803265.168719:0:23940:0:(tgt_grant.c:150:tgt_check_export_grants()) t32fs-OST0000: cli t32fs-MDT0000-mdtlov_UUID/ffff880132ffac00 dirty 0 pend 0 grant 0 00002000:00000020:1.0:1501803265.168720:0:23940:0:(tgt_grant.c:150:tgt_check_export_grants()) t32fs-OST0000: cli t32fs-MDT0001-mdtlov_UUID/ffff880127b2b400 dirty 0 pend 0 grant 0 00002000:00000020:1.0:1501803265.168721:0:23940:0:(tgt_grant.c:144:tgt_check_export_grants()) t32fs-OST0000: processing self export: 20316160 0 0 00002000:00000020:1.0:1501803265.168722:0:23940:0:(tgt_grant.c:150:tgt_check_export_grants()) t32fs-OST0000: cli t32fs-OST0000_UUID/ffff880063d06000 dirty 0 pend 0 grant 20316160 00002000:00000020:1.0:1501803265.168723:0:23940:0:(ofd_obd.c:760:ofd_statfs()) 629 blocks: 172 free, 1 avail; 1836 objects: 1349 free; state 0 00002000:00000001:1.0:1501803265.168724:0:23940:0:(ofd_obd.c:794:ofd_statfs()) Process leaving </pre> </div></div> <p>The OFD said it has granted most of spaces to client, but it seems not to be true:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@centos7 ~]# cat /proc/fs/lustre/osc/t32fs-OST0000-osc-ffff880131f58000/cur_grant_bytes 0 </pre> </div></div> <p>So when it failed, it thought there is no space available at the corresponding OST. However, it seems there are still plenty of space out there:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>UUID 1K-blocks Used Available Use% Mounted on t32fs-MDT0000_UUID 80512 5376 73088 7% /tmp/t32/mnt/lustre[MDT:0] t32fs-MDT0001_UUID 80640 5120 73472 7% /tmp/t32/mnt/lustre[MDT:1] t32fs-OST0000_UUID 80512 58496 19968 75% /tmp/t32/mnt/lustre[OST:0] filesystem_summary: 80512 58496 19968 75% /tmp/t32/mnt/lustre [root@centos7 ~]# zpool list -v t32fs-ost1 NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT t32fs-ost1 190M 57.4M 133M - 26% 30% 1.00x ONLINE - /tmp/t32/ost 190M 57.4M 133M - 26% 30% </pre> </div></div> <p>I guess this would be a BUG about how statfs works on osd-zfs.</p> <p>Alex,</p> <p>Do you have any opinion on this?</p> <p>Thanks.<br/> Joe</p> <p>will try to check locally..</p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28408" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28408</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9826" title="conf-sanity: test 32b failed with 1" class="issue-link" data-issue-key="LU-9826"><del>LU-9826</del></a> tests: Do not run con-sanity 32b with ZFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 91b437c635de5c5421964c40005e49aaa1cdca0e</p> <p> disk2_7-zfs.tar.bz2 seems to be containing a very small filesystem:<br/> Upgrading from disk2_7-zfs.tar.bz2, created with:<br/> NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT<br/> t32fs-mdt1 190M 6.26M 184M - 11% 3% 1.00x ONLINE -<br/> t32fs-ost1 190M 57.7M 132M - 24% 30% 1.00x ONLINE -<br/> Filesystem 1K-blocks Used Available Use% Mounted on<br/> t32fs-ost1/ost1 80512 58496 19968 75% /tmp/t32/mnt/ost<br/> virtual image:<br/> <del>rw</del>------ 1 root root 204800000 Dec 18 2015 ost</p> <p>Upgrading from disk2_4-zfs.tar.bz2, created with:<br/> NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT<br/> t32fs-mdt1 190M 4.50M 186M - - 2% 1.00x ONLINE -<br/> t32fs-ost1 288M 20.0M 268M - - 6% 1.00x ONLINE -<br/> Filesystem 1K-blocks Used Available Use% Mounted on<br/> t32fs-ost1/ost1 146944 19712 125184 14% /tmp/t32/mnt/ost<br/> virtual image:<br/> <del>rw</del>------ 1 root root 307200000 Dec 1 2015 ost</p> <p>i.e. 2.4 disk image is ~100MB larger.</p> <p>also, from what I heard before ZFS-0.7 reserves more space for remove operations. I think it's a possible reason why we can't pass with 2.7 image using ZFS-0.7</p> <p>Alex - do you have a theory to explain why the granted bytes on the OFD doesn't match the number on the client side? I guess that would be the actual problem. </p> <p>because we take a grant for precreates as well.</p> <p>Hello,</p> <p>If recreate the image, how big the disk image is recommended? Besides 2.7, is there any other image needs update?</p> <p>Sarah, 2.4 image is 300MB and the test works fine with that image.</p> <p>Thanks Alex, I will recreate the 2.7 one and update this ticket.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28408/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28408/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9826" title="conf-sanity: test 32b failed with 1" class="issue-link" data-issue-key="LU-9826"><del>LU-9826</del></a> tests: Do not run conf-sanity 32b with ZFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d564becbe58f7fb349cbe19a7f23e447278310d5</p> <p>Assigning to Sarah for the recreation of the 2.7 image.</p> <p>Landed in 2.11</p> <p>Reopening because Sarah is working on a fix/ updating the disk image for ZFS 0.7.0 (probably 0.7.1). </p> <p>Wei Liu (wei3.liu@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28464" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28464</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9826" title="conf-sanity: test 32b failed with 1" class="issue-link" data-issue-key="LU-9826"><del>LU-9826</del></a> test: Recreate disk2_7-zfs image to have a bigger file system<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f2cdd71f3bb9c1885172089e0afd9b534eb5fb4b</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28504" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28504</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9826" title="conf-sanity: test 32b failed with 1" class="issue-link" data-issue-key="LU-9826"><del>LU-9826</del></a> tests: Do not run conf-sanity 32b with ZFS<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 773715574a41b3f57c68f8b31ca61a4e8f9e4b30</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28568" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28568</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9826" title="conf-sanity: test 32b failed with 1" class="issue-link" data-issue-key="LU-9826"><del>LU-9826</del></a> test: Recreate disk2_7-zfs image to have a bigger file system<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 191a90dbc8563204b05fea50a91408d4d95bcccc</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28464/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28464/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9826" title="conf-sanity: test 32b failed with 1" class="issue-link" data-issue-key="LU-9826"><del>LU-9826</del></a> test: Recreate disk2_7-zfs image to have a bigger file system<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6ab9866730c1bd2344869502586fc322f6ed7b71</p> <p>Landed for 2.11</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/28504/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28504/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9826" title="conf-sanity: test 32b failed with 1" class="issue-link" data-issue-key="LU-9826"><del>LU-9826</del></a> tests: Do not run conf-sanity 32b with ZFS<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: ee0b6826d606be8d7edeb2210a87ba27d8ff76cc</p> Related LU-9888 Development Rank 1|hzzhqn: Rank (Obsolete) 9223372036854775807 Severity