https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3752 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/73a518da-029e-11e3-b384-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/73a518da-029e-11e3-b384-52540035b04c</a>.</p> <p>The sub-test test_18 failed with the following error:</p> <blockquote> <p>expect 104857600, got 42991616. Verifying file failed!</p></blockquote> <p>Info required for matching: sanity-quota 18</p> LU-3752

sanity-quota test_18: expect 104857600, got 42991616. Verifying file failed!

Bug Minor Open Unresolved Oleg Drokin Maloo yuc2 Tue, 13 Aug 2013 21:13:45 +0000 Mon, 17 Jul 2017 19:26:01 +0000 Lustre 2.4.1 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.5.1 Lustre 2.8.0 0 7 <p>I just found that this is a regression introduced by the patch in build #28 on Lustre b2_4 branch.</p> <p>Before build #28, sanity-quota test 18 always passed on Lustre b2_4 branch. Since build #28, there were 6 full test runs on build #29 against the RHEL6 and SLES11SP2 clients, 2 test runs hit sanity-quota test 18 failure:</p> <p>Failed test runs:<br/> <a href="https://maloo.whamcloud.com/test_sets/f6c41656-0421-11e3-90ba-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f6c41656-0421-11e3-90ba-52540035b04c</a> (RHEL6)<br/> <a href="https://maloo.whamcloud.com/test_sets/1f547576-0282-11e3-a4b4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/1f547576-0282-11e3-a4b4-52540035b04c</a> (RHEL6)</p> <p>Passed test runs:<br/> <a href="https://maloo.whamcloud.com/test_sets/8a5797c0-0248-11e3-a4b4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8a5797c0-0248-11e3-a4b4-52540035b04c</a> (RHEL6)<br/> <a href="https://maloo.whamcloud.com/test_sets/31d735f2-02b0-11e3-a4b4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/31d735f2-02b0-11e3-a4b4-52540035b04c</a> (SLES11SP2)<br/> <a href="https://maloo.whamcloud.com/test_sets/b109a6b8-0259-11e3-b384-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b109a6b8-0259-11e3-b384-52540035b04c</a> (RHEL6)<br/> <a href="https://maloo.whamcloud.com/test_sets/e4b07626-039f-11e3-9824-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e4b07626-039f-11e3-9824-52540035b04c</a> (RHEL6)</p> <p>On master branch, the test passed on build #1582, and failed on build #1591. The builds between them were not tested. By looking over the patches on these builds and those patches in b2_4 build #28, the following ones are intersections:</p> <p><a href="https://jira.whamcloud.com/browse/LU-3643" title="hsm_restore caused OSS node crash" class="issue-link" data-issue-key="LU-3643"><del>LU-3643</del></a> ofd: get data version only if file exists<br/> <a href="https://jira.whamcloud.com/browse/LU-3585" title="Client panic during IOR single file per process: Lnet out of Memory" class="issue-link" data-issue-key="LU-3585"><del>LU-3585</del></a> ptlrpc: Fix a crash when dereferencing NULL pointer <br/> <a href="https://jira.whamcloud.com/browse/LU-3636" title="HSM restore failed at copy end" class="issue-link" data-issue-key="LU-3636"><del>LU-3636</del></a> llapi: llapi_hsm_copy_end() on correct FID on restore.</p> <p>Oleg</p> <p>Can you please try to further identify the cause of this regression?</p> <p>Thanks</p> <p>Peter</p> <p>A very suspicious common pattern is observed in those test results.</p> <p>All successful test runs have:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d0.sanity-quota/d18/f.sanity-quota.18] [count=100] [oflag=direct] CMD: client-20-ib sync; sync; sync Filesystem 1K-blocks Used Available Use% Mounted on client-20-ib@o2ib:/lustre 14222720 13440 14150272 1% /mnt/lustre </pre> </div></div> <p>All failing runs have:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Write 100M (directio) ... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d0.sanity-quota/d18/f.sanity-quota.18] [count=100] [oflag=direct] CMD: client-26vm7 sync; sync; sync Filesystem 1K-blocks Used Available Use% Mounted on client-26vm7@tcp:/lustre 1464484 264460 1118928 20% /mnt/lustre </pre> </div></div> <p>So, my question is - why newer testruns (that are failing) have 10x less disk space? I bet this is why the test is dying with out of space error now - because striping is also not used and so with previously present files there's just not enough space in the new scheme of things.</p> <p>For failed test runs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>MDSSIZE=1939865 OSTSIZE=223196 </pre> </div></div> <p>For passed test runs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>MDSSIZE=2097152 OSTSIZE=2097152 </pre> </div></div> <p>The real failure was:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>dd: writing `/mnt/lustre/d0.sanity-quota/d18/f.sanity-quota.18': No space left on device </pre> </div></div> <p>We need improve the test script to check available space.</p> <p>space check added<br/> <a href="http://review.whamcloud.com/7366" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7366</a></p> <p>Still leaves open the question of why we're running out of space in the first place.</p> <p>hit this issue in lustre-master build # 1784 <br/> client is running SLES11 SP3</p> <p><a href="https://maloo.whamcloud.com/test_sets/67469a2c-5bbe-11e3-8d79-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/67469a2c-5bbe-11e3-8d79-52540035b04c</a></p> <p>Lustre client build: <a href="http://build.whamcloud.com/job/lustre-b2_4/70/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/70/</a> (2.4.2)<br/> Lustre server build: <a href="http://build.whamcloud.com/job/lustre-b2_5/13/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/13/</a></p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/56bce390-7e7e-11e3-925a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/56bce390-7e7e-11e3-925a-52540035b04c</a></p> <p>Hit this failure in lustre-master tag-2.5.57(build # 1945) testing for zfs:<br/> <a href="https://maloo.whamcloud.com/test_sessions/e29960fc-b031-11e3-9bc4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/e29960fc-b031-11e3-9bc4-52540035b04c</a></p> <p>In the previous build 1944 and 1943, this test passed:<br/> <a href="https://maloo.whamcloud.com/test_sessions/28690920-af2e-11e3-bac7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/28690920-af2e-11e3-bac7-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sessions/1af5c146-ae6d-11e3-a4ae-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/1af5c146-ae6d-11e3-a4ae-52540035b04c</a></p> <p>hit this on current master build#3305 RHEL6.7 zfs<br/> <a href="https://testing.hpdd.intel.com/test_sets/91fc1ebc-bc84-11e5-b3b7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/91fc1ebc-bc84-11e5-b3b7-5254006e85c2</a></p> <p>Encountered another instance for FULL - EL6.7 Server/EL6.7 Client - ZFS , master , build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/9e6de21c-cb47-11e5-a59a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9e6de21c-cb47-11e5-a59a-5254006e85c2</a></p> <p>Another instance on master for FULL - EL7.1 Server/EL7.1 Client - ZFS, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/e109a106-cb88-11e5-b49e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e109a106-cb88-11e5-b49e-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 - EL6.7 Server/EL6.7 Client - ZFS, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/9e6de21c-cb47-11e5-a59a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9e6de21c-cb47-11e5-a59a-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 -EL7.1 Server/EL7.1 Client - ZFS, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/e109a106-cb88-11e5-b49e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e109a106-cb88-11e5-b49e-5254006e85c2</a></p> Development Rank 1|hzvxwn: Rank (Obsolete) 9677 Severity