Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-17251] parallel-scale test_rr_alloc: max/min OST objects (2800 : 923) too different https://jira.whamcloud.com/browse/LU-17251 Lustre <h2><a name="parallelscaletestrralloc%3Amax%2FminOSTobjects%282800%3A923%29toodifferent"></a>parallel-scale test_rr_alloc: max/min OST objects (2800 : 923) too different</h2> <p>RHEL 8.8 x86_64 master/2.15.58.130</p> <p>Failed session: <a href="https://testing.whamcloud.com/test_sessions/cde52bbc-3bcd-40b6-b9ad-fa35f8bc4deb" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/cde52bbc-3bcd-40b6-b9ad-fa35f8bc4deb</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> CMD: onyx-82vm9 /usr/sbin/lctl set_param -n lod.lustre-MDT*.qos_threshold_rr=100 osp.lustre-OST*-osc-MDT*.create_count=3052 CMD: onyx-82vm10 /usr/sbin/lctl set_param -n lod.lustre-MDT*.qos_threshold_rr=100 osp.lustre-OST*-osc-MDT*.create_count=3052 CMD: onyx-82vm9 /usr/sbin/lctl set_param -n lod.lustre-MDT*.qos_threshold_rr=100 osp.lustre-OST*-osc-MDT*.create_count=3052 CMD: onyx-82vm10 /usr/sbin/lctl set_param -n lod.lustre-MDT*.qos_threshold_rr=100 osp.lustre-OST*-osc-MDT*.create_count=3052 CMD: onyx-82vm9 /usr/sbin/lctl get_param -n debug CMD: onyx-82vm10,onyx-82vm1.onyx.whamcloud.com,onyx-82vm2,onyx-82vm5,onyx-82vm9 /usr/sbin/lctl set_param -n debug=0 CMD: onyx-82vm1.onyx.whamcloud.com,onyx-82vm2 /usr/sbin/lctl set_param debug=\<span class="code-quote">"<span class="code-keyword">super</span> ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck\"</span> CMD: onyx-82vm10,onyx-82vm5,onyx-82vm9 /usr/sbin/lctl set_param debug=\<span class="code-quote">"<span class="code-keyword">super</span> ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck\"</span> CMD: onyx-82vm9 /usr/sbin/lctl get_param -n debug CMD: onyx-82vm10,onyx-82vm1.onyx.whamcloud.com,onyx-82vm2,onyx-82vm5,onyx-82vm9 /usr/sbin/lctl set_param -n debug=0 - unlinked 0 (time 1698827219 ; total 0 ; last 0) total: 1032 unlinks in 1 seconds: 1032.000000 unlinks/second CMD: onyx-82vm1.onyx.whamcloud.com,onyx-82vm2 /usr/sbin/lctl set_param debug=\<span class="code-quote">"<span class="code-keyword">super</span> ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck\"</span> CMD: onyx-82vm10,onyx-82vm5,onyx-82vm9 /usr/sbin/lctl set_param debug=\<span class="code-quote">"<span class="code-keyword">super</span> ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck\"</span> CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_last_id CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0000-osc-MDT0000.prealloc_next_id Warning: test may fail from too few objs on OST0 CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0001-osc-MDT0000.prealloc_last_id CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0001-osc-MDT0000.prealloc_next_id Warning: test may fail from too few objs on OST1 CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0002-osc-MDT0000.prealloc_last_id CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0002-osc-MDT0000.prealloc_next_id CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0003-osc-MDT0000.prealloc_last_id CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0003-osc-MDT0000.prealloc_next_id CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0004-osc-MDT0000.prealloc_last_id CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0004-osc-MDT0000.prealloc_next_id Warning: test may fail from too few objs on OST4 CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0005-osc-MDT0000.prealloc_last_id CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0005-osc-MDT0000.prealloc_next_id Warning: test may fail from too few objs on OST5 CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0006-osc-MDT0000.prealloc_last_id CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0006-osc-MDT0000.prealloc_next_id Warning: test may fail from too few objs on OST6 CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0007-osc-MDT0000.prealloc_last_id CMD: onyx-82vm9 lctl get_param -n osp.lustre-OST0007-osc-MDT0000.prealloc_next_id Warning: test may fail from too few objs on OST7 + chmod 0777 /mnt/lustre drwxrwxrwx 4 root root 4096 Nov 1 08:26 /mnt/lustre + su mpiuser bash -c <span class="code-quote">"/usr/lib64/openmpi/bin/mpirun --mca btl tcp,self --mca btl_tcp_if_include eth0 -mca boot ssh --oversubscribe -np 22 /usr/lib64/openmpi/bin/rr_alloc /tmp/rr_alloc_mntpt/lustre/drr_alloc.parallel-scale/f 555 2 "</span> CMD: onyx-82vm9 /usr/sbin/lctl set_param -n lod.lustre-MDT0000-mdtlov.qos_threshold_rr=17% CMD: onyx-82vm9 /usr/sbin/lctl set_param -n osp.lustre-OST0000-osc-MDT0000.create_count=128 CMD: onyx-82vm9 /usr/sbin/lctl set_param -n osp.lustre-OST0001-osc-MDT0000.create_count=64 CMD: onyx-82vm9 /usr/sbin/lctl set_param -n osp.lustre-OST0002-osc-MDT0000.create_count=64 CMD: onyx-82vm9 /usr/sbin/lctl set_param -n osp.lustre-OST0003-osc-MDT0000.create_count=64 CMD: onyx-82vm9 /usr/sbin/lctl set_param -n osp.lustre-OST0004-osc-MDT0000.create_count=128 CMD: onyx-82vm9 /usr/sbin/lctl set_param -n osp.lustre-OST0005-osc-MDT0000.create_count=128 CMD: onyx-82vm9 /usr/sbin/lctl set_param -n osp.lustre-OST0006-osc-MDT0000.create_count=128 CMD: onyx-82vm9 /usr/sbin/lctl set_param -n osp.lustre-OST0007-osc-MDT0000.create_count=64 parallel-scale test_rr_alloc: @@@@@@ FAIL: max/min OST objects (2230 : 1144) too different Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:6727:error() = /usr/lib64/lustre/tests/functions.sh:1133:run_rr_alloc() = /usr/lib64/lustre/tests/parallel-scale.sh:163:test_rr_alloc() = /usr/lib64/lustre/tests/test-framework.sh:7067:run_one() = /usr/lib64/lustre/tests/test-framework.sh:7123:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:6953:run_test() = /usr/lib64/lustre/tests/parallel-scale.sh:165:main() </pre> </div></div> LU-17251 parallel-scale test_rr_alloc: max/min OST objects (2800 : 923) too different Bug Minor In Progress Unresolved Alex Deiter Alex Deiter Wed, 1 Nov 2023 22:10:03 +0000 Wed, 20 Dec 2023 01:57:47 +0000 Lustre 2.16.0 0 3 <p>"Alex Deiter &lt;alex.deiter@gmail.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52940" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52940</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17251" title="parallel-scale test_rr_alloc: max/min OST objects (2800 : 923) too different" class="issue-link" data-issue-key="LU-17251">LU-17251</a> test: check for OST precreated objects<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2c6d1107f9c05c214022743945f74f777ad2bbf5</p> <p>I think this is also the same as <a href="https://jira.whamcloud.com/browse/LU-14377" title="parallel-scale test rr_alloc fails with &#39;&#39;Uneven distribution detected: difference between maximum files per OST (1528) and minimum files per OST (1525) must not be greater than 2&#39;&#39;" class="issue-link" data-issue-key="LU-14377"><del>LU-14377</del></a>, but just the message changed due to added debugging for the test failures in patch <a href="https://review.whamcloud.com/48914" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48914</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-14377" title="parallel-scale test rr_alloc fails with &#39;&#39;Uneven distribution detected: difference between maximum files per OST (1528) and minimum files per OST (1525) must not be greater than 2&#39;&#39;" class="issue-link" data-issue-key="LU-14377"><del>LU-14377</del></a> tests: make parallel-scale/rr_alloc less strict</tt>".</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52968" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52968</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17251" title="parallel-scale test_rr_alloc: max/min OST objects (2800 : 923) too different" class="issue-link" data-issue-key="LU-17251">LU-17251</a> osp: force precreate if create_count grows<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0c4611acf569dee51e2aa3bf4ee700eeffb208f1</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=Deiter" class="user-hover" rel="Deiter">Deiter</a> I think my patch is complementary to yours. Yours is improving the test script, and the wait loop is still needed since my patch does not wait for the precreates to finish before returning from set_param. However, the "createmany/unlinkmany" dance is no longer needed, and maybe never was needed, and is counter-productive in my opinion.</p> <p>Hello <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a>,</p> <p>Thank you very much for the patch and detailed explanation!<br/> Please let me update my patch!</p> <p>Thank you!</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52968/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52968/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17251" title="parallel-scale test_rr_alloc: max/min OST objects (2800 : 923) too different" class="issue-link" data-issue-key="LU-17251">LU-17251</a> osp: force precreate if create_count grows<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: df5b4c0a8b076f36c63da89d81fe020ca29d39aa</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53245" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53245</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17251" title="parallel-scale test_rr_alloc: max/min OST objects (2800 : 923) too different" class="issue-link" data-issue-key="LU-17251">LU-17251</a> osp: start OST object precreate earlier<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6ffb849d7086a2b2ae48f274d4f5b1b8fbf83fe2</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53245/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53245/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17251" title="parallel-scale test_rr_alloc: max/min OST objects (2800 : 923) too different" class="issue-link" data-issue-key="LU-17251">LU-17251</a> osp: start OST object precreate earlier<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0ccf1311382059d22cf4788136939647fba1317a</p> Related LU-14377 LU-13941 Development Rank 1|i040cf: Rank (Obsolete) 9223372036854775807 Severity