https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16541 Lustre <p>This issue was created by maloo for Arshad &lt;arshad.hussain@aeoncomputing.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/a0c04b66-87b8-4b06-aee8-8ae97f9e229d" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/a0c04b66-87b8-4b06-aee8-8ae97f9e229d</a></p> <p>test_64f failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>buffered io, not write rpc: grants mismatch: 12656640, expected 4218880 </pre> </div></div> <p>Test session details:<br/> clients: <a href="https://build.whamcloud.com/job/lustre-reviews/92173" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/92173</a> - 4.18.0-372.32.1.el8_6.aarch64<br/> servers: <a href="https://build.whamcloud.com/job/lustre-reviews/92173" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/92173</a> - 4.18.0-348.23.1.el8_lustre.x86_64</p> <p>&lt;&lt;Please provide additional information about the failure here&gt;&gt;</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_64f - buffered io, not write rpc: grants mismatch: 12656640, expected 4218880</p> LU-16541

sanity test_64f: buffered io, not write rpc: grants mismatch: 12656640, expected 4218880

Bug Critical Resolved Fixed Patrick Farrell Maloo arm Thu, 9 Feb 2023 02:18:23 +0000 Wed, 20 Dec 2023 04:04:14 +0000 Fri, 25 Aug 2023 20:24:24 +0000 Lustre 2.16.0 Lustre 2.16.0 Lustre 2.15.4 0 11 <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/3612a72a-bde4-4fcd-9cb9-4eb4ed7ceab8" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3612a72a-bde4-4fcd-9cb9-4eb4ed7ceab8</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/6649c5ad-a4db-40ba-abe4-67822a7227c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/6649c5ad-a4db-40ba-abe4-67822a7227c2</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/e4028aaf-757e-46c5-9ea9-440a7bda4e21" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e4028aaf-757e-46c5-9ea9-440a7bda4e21</a></p> <p>Sadly its not just  ARM. Looking for grant mismatch you will a bunch of duplicate tickets.</p> <p>Patrick, could you please take a look at this. This subtest is now the top cause of failures, when it previously was only failing on aarch64. Since it is running as part of sanity, this subtest is run 9x per patch review test (unless run with 'trivial') so with a failure rate around 1/16 runs it is almost guaranteed to affect every patch.</p> <p>it would make sense to run test result searches on a per-week basis to see if you can identify when the subtest first started failing on x86, and then use that to identify culprit patches that landed in that time period:</p> <p><a href="https://testing.whamcloud.com/search?client_branch_type_id=24a6947e-04a9-11e1-bb5f-52540025f9af&amp;horizon=518400&amp;status%5B%5D=FAIL&amp;test_set_script_id=f9516376-32bc-11e0-aaee-52540025f9ae&amp;sub_test_script_id=2a009de0-e0af-11e8-89f8-52540065bddc&amp;source=sub_tests#redirect" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/search?client_branch_type_id=24a6947e-04a9-11e1-bb5f-52540025f9af&amp;horizon=518400&amp;status%5B%5D=FAIL&amp;test_set_script_id=f9516376-32bc-11e0-aaee-52540025f9ae&amp;sub_test_script_id=2a009de0-e0af-11e8-89f8-52540065bddc&amp;source=sub_tests#redirect</a></p> <p>Doing a quick search showed that all of the failures on master in the week of 2023-04-16 were for your <a href="https://jira.whamcloud.com/browse/LU-13805" title="i/o path: Unaligned direct i/o" class="issue-link" data-issue-key="LU-13805">LU-13805</a> unaligned DIO patch series at that time, but none of those patches have landed, unless something was split out into a separate patch. However, it may be possible to do some differential analysis between the start of the failures on master vs. b_es6_0 to see when particular patches landed to each branch.</p> <p>Yeah, I actually started running this locally and couldn't reproduce, but I'm happy to give this a try.  I will see if I can figure something out from the landings, but I may also just go directly at the bug as well.</p> <p> <span class="image-wrap" style=""><img src="https://jira.whamcloud.com/secure/attachment/50100/50100_Screen+Shot+2023-08-21+at+14.07.18.png" style="border: 0px solid black" /></span></p> <p>It looks like the spike in subtest failures on master started on 2023-08-02.</p> <p>Ah, thank you.</p> <p>"Patrick Farrell &lt;pfarrell@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52022" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52022</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16541" title="sanity test_64f: buffered io, not write rpc: grants mismatch: 12656640, expected 4218880 " class="issue-link" data-issue-key="LU-16541"><del>LU-16541</del></a> tests: bisect<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9a360210bff4e076ae59171fe32c3656184a5901</p> <p>"Patrick Farrell &lt;pfarrell@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52023" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52023</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16541" title="sanity test_64f: buffered io, not write rpc: grants mismatch: 12656640, expected 4218880 " class="issue-link" data-issue-key="LU-16541"><del>LU-16541</del></a> tests: bisect<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a042bf7d62ec99b8b6cc299be998408286dd2b2a</p> <p>he other annoyance with this test failure btw is if it hits, it takes 1 hour for the test to finish which makes it a timeout on janitor:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sanity.test_64f.test_log.oleg346-client.log2023-08-19 04:46 1.6K  sanity.test_64g.test_log.oleg346-client.log2023-08-19 05:46 1.6K </pre> </div></div> <p>Example here, but they ar eall like this. I wonder if the unbounded wait just waits for something else that happens to run for an hour? Nothing obvious in the test output:</p> <p><a href="http://testing.linuxhacker.ru/lustre-reports/33659/testresults/sanity2-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/" class="external-link" target="_blank" rel="nofollow noopener">http://testing.linuxhacker.ru/lustre-reports/33659/testresults/sanity2-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/</a></p> <p>the one hour duration also holds for the maloo</p> <p>"Patrick Farrell &lt;pfarrell@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52040" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52040</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16541" title="sanity test_64f: buffered io, not write rpc: grants mismatch: 12656640, expected 4218880 " class="issue-link" data-issue-key="LU-16541"><del>LU-16541</del></a> tests: Improve test 64f<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 203bdc67a292bb1daff50d379d7dbfe5a86cc89c</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52040/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52040/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16541" title="sanity test_64f: buffered io, not write rpc: grants mismatch: 12656640, expected 4218880 " class="issue-link" data-issue-key="LU-16541"><del>LU-16541</del></a> tests: Improve test 64f<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 33e4d86a480b860e0a3b4b51c7c6da6ec0159e51</p> <p>Merged for 2.16</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52096" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52096</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16541" title="sanity test_64f: buffered io, not write rpc: grants mismatch: 12656640, expected 4218880 " class="issue-link" data-issue-key="LU-16541"><del>LU-16541</del></a> tests: Improve test 64f<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 52d69926bb180b4d8e7ebf30f5d27431da544c17</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52096/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52096/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16541" title="sanity test_64f: buffered io, not write rpc: grants mismatch: 12656640, expected 4218880 " class="issue-link" data-issue-key="LU-16541"><del>LU-16541</del></a> tests: Improve test 64f<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: 478ca310b4204b4354245e6261edae6fef0ae497</p> Related LU-16673 Development Rank 1|i03ce7: Rank (Obsolete) 9223372036854775807 Severity