https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9096 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;andreas.dilger@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/b06b5a90-e8f0-11e6-935d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b06b5a90-e8f0-11e6-935d-5254006e85c2</a>.</p> <p>The sub-test test_253 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: trevis-43vm3 /usr/sbin/lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_status prealloc_status -28 dd: failed to open '/mnt/lustre/d253.sanity/2': No space left on device 5+0 records in 5+0 records out 5242880 bytes (5.2 MB) copied, 0.0534031 s, 98.2 MB/s CMD: trevis-43vm3 /usr/sbin/lctl set_param -n osd*.*MD*.force_sync 1 CMD: trevis-43vm3 /usr/sbin/lctl get_param -n osc.*MDT*.sync_* : : CMD: trevis-43vm3 /usr/sbin/lctl get_param -n osc.*MDT*.sync_* Delete is not completed in 28 seconds Waiting for local destroys to complete CMD: trevis-43vm3 lctl get_param -n lov.*.qos_maxage sanity test_253: @@@@@@ FAIL: File creation failed after rm </pre> </div></div> <p>The MDS console log shows that there are no precreated objects for this OST:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>20:44:10:[11571.982460] LustreError: 22503:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x20000234c:0x13b8:0x0]: have 0 want 1 </pre> </div></div> <p>Info required for matching: sanity 253</p> LU-9096

sanity test_253: File creation failed after rm

Bug Critical Open Unresolved WC Triage Maloo always_except zfs Thu, 9 Feb 2017 17:54:00 +0000 Mon, 22 Jan 2024 23:40:11 +0000 Lustre 2.9.0 Lustre 2.10.0 Lustre 2.14.0 0 14 <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>20:45:07:[ 4933.488092] Lustre: DEBUG MARKER: == sanity test 253: Check object allocation limit ==================================================== 20:39:14 (1484858354) 20:45:07:[ 5040.092189] INFO: task txg_sync:20227 blocked for more than 120 seconds. 20:45:07:[ 5040.096367] txg_sync D 0000000000000001 0 20227 2 0x00000080 20:45:07:[ 5040.106578] Call Trace: 20:45:07:[ 5040.108189] [&lt;ffffffff8168b9e9&gt;] schedule+0x29/0x70 20:45:07:[ 5040.110193] [&lt;ffffffff81689439&gt;] schedule_timeout+0x239/0x2d0 20:45:07:[ 5040.112031] [&lt;ffffffff810c4ff2&gt;] ? default_wake_function+0x12/0x20 20:45:07:[ 5040.114129] [&lt;ffffffff810ba238&gt;] ? __wake_up_common+0x58/0x90 20:45:07:[ 5040.115970] [&lt;ffffffff81060c1f&gt;] ? kvm_clock_get_cycles+0x1f/0x30 20:45:07:[ 5040.118039] [&lt;ffffffff8168af8e&gt;] io_schedule_timeout+0xae/0x130 20:45:07:[ 5040.119922] [&lt;ffffffff810b1416&gt;] ? prepare_to_wait_exclusive+0x56/0x90 20:45:07:[ 5040.122039] [&lt;ffffffff8168b028&gt;] io_schedule+0x18/0x20 20:45:07:[ 5040.123883] [&lt;ffffffffa0662617&gt;] cv_wait_common+0xa7/0x130 [spl] 20:45:07:[ 5040.125886] [&lt;ffffffff810b1720&gt;] ? wake_up_atomic_t+0x30/0x30 20:45:07:[ 5040.127688] [&lt;ffffffffa06626f8&gt;] __cv_wait_io+0x18/0x20 [spl] 20:45:07:[ 5040.129716] [&lt;ffffffffa07bc51b&gt;] zio_wait+0x10b/0x1f0 [zfs] 20:45:07:[ 5040.131471] [&lt;ffffffffa0745cdf&gt;] dsl_pool_sync+0xbf/0x440 [zfs] 20:45:07:[ 5040.133451] [&lt;ffffffffa0760868&gt;] spa_sync+0x388/0xb50 [zfs] 20:45:07:[ 5040.135159] [&lt;ffffffff810b174b&gt;] ? autoremove_wake_function+0x2b/0x40 20:45:07:[ 5040.137112] [&lt;ffffffff81689b92&gt;] ? mutex_lock+0x12/0x2f 20:45:07:[ 5040.138815] [&lt;ffffffffa07724e5&gt;] txg_sync_thread+0x3c5/0x620 [zfs] 20:45:07:[ 5040.142523] [&lt;ffffffffa065d851&gt;] thread_generic_wrapper+0x71/0x80 [spl] 20:45:07:[ 5040.146352] [&lt;ffffffff810b064f&gt;] kthread+0xcf/0xe0 </pre> </div></div> <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/7029362e-0e0b-11e7-9053-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7029362e-0e0b-11e7-9053-5254006e85c2</a></p> <p>+1 on master<br/> <a href="https://testing.hpdd.intel.com/sub_tests/987b39ae-0eee-11e7-b5b2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/987b39ae-0eee-11e7-b5b2-5254006e85c2</a></p> <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/c0ddafc8-f0b9-4bd4-b3d8-bee81935e304" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/c0ddafc8-f0b9-4bd4-b3d8-bee81935e304</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/abcc08e0-15cd-11e7-9073-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/abcc08e0-15cd-11e7-9073-5254006e85c2</a></p> <p>more on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/8bf82420-1769-11e7-9073-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8bf82420-1769-11e7-9073-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/485d3906-1973-11e7-b742-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/485d3906-1973-11e7-b742-5254006e85c2</a></p> <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/fdc0b50c-2067-11e7-9073-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fdc0b50c-2067-11e7-9073-5254006e85c2</a></p> <p>Nathaniel</p> <p>It seems like this test failure only impacts ZFS runs</p> <p>Peter</p> <p>Alex, you have a patch <a href="https://review.whamcloud.com/18341" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/18341</a> that may be related to this problem? </p> <p>Another on master:</p> <p><a href="https://testing.hpdd.intel.com/test_sessions/10710b40-d31b-41be-8e9a-07f9c3aec583" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/10710b40-d31b-41be-8e9a-07f9c3aec583</a></p> <p>Andreas, that patch prevents indefinit blocking which isn't the case (the tests completed) ?</p> <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/af75a44a-28e4-11e7-9073-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/af75a44a-28e4-11e7-9073-5254006e85c2</a></p> <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/cb756bb2-2a97-11e7-9de9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cb756bb2-2a97-11e7-9de9-5254006e85c2</a></p> <p>+1 on master :<a href="https://testing.hpdd.intel.com/test_sets/34f1e788-2aab-11e7-8920-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/34f1e788-2aab-11e7-8920-5254006e85c2</a></p> <p>updated statfs: rc = 0, state, ffree 57923, bavail 52864, avail 206, low 1<br/> updated statfs: rc = 0, state, ffree 57923, bavail 52864, avail 206, low 1<br/> updated statfs: rc = 0, state enospc, ffree 57923, bavail 39552, avail 154, low 209<br/> updated statfs: rc = 0, state enospc, ffree 18345, bavail 44512, avail 173, low 209<br/> updated statfs: rc = 0, state enospc, ffree 18345, bavail 44512, avail 173, low 209<br/> updated statfs: rc = 0, state enospc, ffree 18345, bavail 44512, avail 173, low 209<br/> updated statfs: rc = 0, state enospc, ffree 18345, bavail 44512, avail 173, low 209<br/> updated statfs: rc = 0, state enospc, ffree 18345, bavail 42848, avail 167, low 209<br/> updated statfs: rc = 0, state enospc, ffree 18345, bavail 42848, avail 167, low 209<br/> updated statfs: rc = 0, state enospc, ffree 57908, bavail 52864, avail 206, low 209<br/> updated statfs: rc = 0, state enospc, ffree 57908, bavail 52864, avail 206, low 209</p> <p>sanity/253 sets low to 209:<br/> local blocks=$($LFS df $MOUNT | grep $ost_name | awk '</p> { print $4 } <p>')<br/> local new_lwm=$((blocks/1024-10))</p> <p>osp.lustre-OST0000-osc-MDT0000.reserved_mb_low=209</p> <p>@@ -14318,7 +14318,7 @@ test_253() {<br/> local blocks=$($LFS df $MOUNT | grep $ost_name | awk '</p> { print $4 } <p>')<br/> echo "OST still has $((blocks/1024)) mbytes free"</p> <ul class="alternate" type="square"> <li>local new_lwm=$((blocks/1024-10))<br/> + local new_lwm=$((blocks/1024-20))<br/> do_facet $SINGLEMDS $LCTL set_param \<br/> osp.$mdtosc_proc1.reserved_mb_high=$((new_lwm+5))<br/> do_facet $SINGLEMDS $LCTL set_param \</li> </ul> <p>solves the problem.. I'm wondering where 10 comes from.</p> <p>&gt; solves the problem.. I'm wondering where 10 comes from.</p> <p>I think it matches the 10 used a few lines before:<br/> dd if=/dev/zero of=$DIR/$tdir/0 bs=1M count=10</p> <p>In the interim it would be good to dump all the osp params when this fails (rather than just prealloc_status).</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=bzzz" class="user-hover" rel="bzzz">bzzz</a>,</p> <p>Instead of hard coding a constant, would it make sense to use <tt>test_framework.sh::fs_log_size()</tt> to ensure there's enough room for the fs log?</p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/27013" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27013</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9096" title="sanity test_253: File creation failed after rm" class="issue-link" data-issue-key="LU-9096">LU-9096</a> test: add sanity 253 to ALWAYS_EXCEPT<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d62f40a8af0fa7d4041517e81a0183bb051c1785</p> <p>Since sanity 253 is failing intermittently and the correctness of the test is being questioned, I've uploaded a patch to add the test to the ALWAYS_EXCEPT list and, thus, not be run.</p> <p>Patch <a href="https://review.whamcloud.com/27013" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27013</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/27013/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27013/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9096" title="sanity test_253: File creation failed after rm" class="issue-link" data-issue-key="LU-9096">LU-9096</a> test: add sanity 253 to ALWAYS_EXCEPT<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: cdc7b3bb16537fc513a64b4f20824b9832efcdf9</p> <p>The test is no longer being run so I will remove the fixversion from this ticket and it can be reintroduced when the test has been given a clean bill of health</p> <p>For some reason this test is now running again and failing intermittently (about 4x per month).</p> <p>It looks like this test was re-enabled by patch <a href="https://review.whamcloud.com/33778" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33778</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-10070" title="PFL self-extending file layout" class="issue-link" data-issue-key="LU-10070"><del>LU-10070</del></a> tests: New test-framework functionality</tt>". Since it currently is not failing very offer, I'm not in a hurry to turn it off again, especially if it is adding coverage for SEL. </p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sessions/c590176e-852a-4080-abb9-e601015fbed9" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/c590176e-852a-4080-abb9-e601015fbed9</a></p> <p>Patch <a href="https://review.whamcloud.com/51861" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/51861</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-15553" title="replay-vbr test 12a fails with &#39;test_12a failed with 4&#39;" class="issue-link" data-issue-key="LU-15553">LU-15553</a> test: mkdir_on_mdt0 in sanity</tt>" is adding this subtest to the <tt>always_except</tt> list again.</p> Duplicate LU-9450 LU-9454 Related LU-7754 LU-15553 LU-10070 LU-9457 Development Rank 1|hzz3cv: Rank (Obsolete) 9223372036854775807 Severity