https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16827 Lustre <p>This issue was created by maloo for Chris Horn &lt;chris.horn@hpe.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/cf8d9cb0-9efd-4978-b9c4-f32417b78f26" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/cf8d9cb0-9efd-4978-b9c4-f32417b78f26</a></p> <p>Test session details:<br/> clients: <a href="https://build.whamcloud.com/job/lustre-reviews/94764" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/94764</a> - 4.18.0-425.10.1.el8_7.x86_64<br/> servers: <a href="https://build.whamcloud.com/job/lustre-reviews/94764" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/94764</a> - 4.18.0-425.10.1.el8_lustre.x86_64</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== obdfilter-survey test 1a: Object Storage Targets survey ========================================================== 21:29:16 (1683754156) + NETTYPE=tcp thrlo=2 nobjhi=1 thrhi=4 size=1024 case=disk rslt_loc=/tmp targets="lustre-OST0000 lustre-OST0001 lustre-OST0002 lustre-OST0003 lustre-OST0004 lustre-OST0005 lustre-OST0006" /usr/bin/obdfilter-survey OST lustre-OST0000 not setup /usr/bin/iokit-libecho: line 221: 420704 Killed remote_shell $host "vmstat 5 &gt;&gt; $host_vmstatf" &amp;&gt; /dev/null program exited with error </pre> </div></div> LU-16827

obdfilter-survey: /usr/bin/obdfilter-survey failed: 2

Bug Major Resolved Fixed Arshad Hussain Maloo Fri, 12 May 2023 18:29:31 +0000 Wed, 29 Nov 2023 22:46:01 +0000 Wed, 29 Nov 2023 22:46:01 +0000 Lustre 2.16.0 Lustre 2.16.0 0 8 <p>"Arshad Hussain &lt;arshad.hussain@aeoncomputing.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51035" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51035</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16827" title="obdfilter-survey: /usr/bin/obdfilter-survey failed: 2" class="issue-link" data-issue-key="LU-16827"><del>LU-16827</del></a> obdfilter: Test patch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 35bbcad1b9e68d5772107a034974878e3ab8cbc5</p> <p>Failure is due to not being able to setup echo_client. <b>lctl dl</b> does not have an entry like below...</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>28 UP echo_client lustreT-OST0000_ecc lustreT-OST0000_ecc_UUID 2</pre> </div></div> <p>Lookup for the above entry is returned false/empty by get_ec_devno()  and cleanup() is called with non-zero value.</p> <p>It looks like obdfilter-survey is failing 100% of test runs now, and it was caused by the landing of patch <a href="https://review.whamcloud.com/50096" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/50096</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13340" title="add LCFG_ADD_UUIDv6 and related commands" class="issue-link" data-issue-key="LU-13340"><del>LU-13340</del></a> lustre: Support large nids in LCFG_ADD_UUID</tt>". Looking back at the test results, it shows that patch 50096 and 3 others based on it at the time (50099, 50102, 50103) <a href="https://testing.whamcloud.com/search?client_branch_type_id=24a6947e-04a9-11e1-bb5f-52540025f9af&amp;status%5B%5D=TIMEOUT&amp;test_set_script_id=11a69f28-4a54-11e0-a7f6-52540025f9af&amp;sub_test_script_id=12143074-4a54-11e0-a7f6-52540025f9af&amp;start_date=2023-05-01&amp;end_date=2023-06-01&amp;source=sub_tests#redirect" class="external-link" target="_blank" rel="nofollow noopener">all failed on 2023-05-02 before they landed</a>, <a href="https://testing.whamcloud.com/search?client_branch_type_id=24a6947e-04a9-11e1-bb5f-52540025f9af&amp;status%5B%5D=TIMEOUT&amp;test_set_script_id=11a69f28-4a54-11e0-a7f6-52540025f9af&amp;sub_test_script_id=12143074-4a54-11e0-a7f6-52540025f9af&amp;start_date=2023-06-01&amp;end_date=2023-07-01&amp;source=sub_tests#redirect" class="external-link" target="_blank" rel="nofollow noopener">more failures on 2023-06-07 and 2023-06-22 when the patch was refreshed</a>, and then <a href="https://testing.whamcloud.com/search?client_branch_type_id=24a6947e-04a9-11e1-bb5f-52540025f9af&amp;test_set_script_id=11a69f28-4a54-11e0-a7f6-52540025f9af&amp;sub_test_script_id=12143074-4a54-11e0-a7f6-52540025f9af&amp;start_date=2023-07-01&amp;end_date=2023-08-01&amp;source=sub_tests#redirect" class="external-link" target="_blank" rel="nofollow noopener">started failing solidly when the patch landed on 2023-07-08</a>.</p> <p> </p> <p>I haven't looked into <b>why</b> this patch caused the failures, but based on Arshad's comments above it may be related to a change in the output format for "<tt>lctl dl</tt>", but I'm not sure.</p> <p>Andreas,  I have the fix (I think) - Need a cleanup (I will get this done) - <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51035" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51035</a></p> <p>The reason  for not getting +1 from Maloo seems to be - <a href="https://jira.whamcloud.com/browse/LU-16861" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-16861</a>.  So along with above patch , <a href="https://jira.whamcloud.com/browse/LU-16861" title="Janitor Testing Fails to copy latest obdfilter-survey (Uses old obdfilter-survey)" class="issue-link" data-issue-key="LU-16861"><del>LU-16861</del></a> must be fixed first according to me.</p> <p>This seems to be fail Failing since May,2023 - I first encounter this on <a href="https://testing.whamcloud.com/test_sessions/79ec0895-a03c-4828-8dc8-c48b959e0990" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/79ec0895-a03c-4828-8dc8-c48b959e0990</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Andreas,  I have the fix (I think) - Need a cleanup (I will get this done) - https://review.whamcloud.com/c/fs/lustre-release/+/51035</pre> </div></div> <p>Done.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51035/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51035/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16827" title="obdfilter-survey: /usr/bin/obdfilter-survey failed: 2" class="issue-link" data-issue-key="LU-16827"><del>LU-16827</del></a> obdfilter: Fix obdfilter-survery/1a<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 91a3b286ba57bb491b5c17600d7cec9e516a428f</p> <p>Looks to be landed for 2.16</p> <p>"Vitaliy Kuznetsov &lt;vkuznetsov@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53083" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53083</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16827" title="obdfilter-survey: /usr/bin/obdfilter-survey failed: 2" class="issue-link" data-issue-key="LU-16827"><del>LU-16827</del></a> obdfilter: Fix "emfperf obdfilter-survey" error<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 753faeec133f0dbb63bb990432a7d80544e21f3e</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53083/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53083/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16827" title="obdfilter-survey: /usr/bin/obdfilter-survey failed: 2" class="issue-link" data-issue-key="LU-16827"><del>LU-16827</del></a> obdfilter: Fix "emfperf obdfilter-survey" error<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6b514c0bd0e3f83cfb9f547148b6fc183db7c4c9</p> <p>Landed for 2.16</p> Duplicate Gantt End to Start LU-16834 LU-16861 Related LU-13340 Development Rank 1|i03lb3: Rank (Obsolete) 9223372036854775807 Severity