https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11597 Lustre <p>sanity test 241b failed with direct I/O:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 241b: dio vs dio ====================================================================== 08:45:10 (1540543510) 1+0 records in 1+0 records out 40960 bytes (41 kB) copied, 0.00274397 s, 14.9 MB/s -rw-r--r-- 1 root root 40960 Oct 26 08:45 /mnt/lustre/f241b.sanity sanity test_241b: @@@@@@ FAIL: test_241b failed with 1 </pre> </div></div> <p>Maloo report: <a href="https://testing.whamcloud.com/test_sets/88bbf5c2-d9d0-11e8-b46b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/88bbf5c2-d9d0-11e8-b46b-52540065bddc</a></p> <p>sanity tests 270a, 315, and sanityn test 16a also failed with the same issue:<br/> <a href="https://testing.whamcloud.com/test_sets/88bbf5c2-d9d0-11e8-b46b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/88bbf5c2-d9d0-11e8-b46b-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/8b2f4282-d9d0-11e8-b46b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8b2f4282-d9d0-11e8-b46b-52540065bddc</a></p> Lustre Build: <a href="https://build.whamcloud.com/job/lustre-master/3811">https://build.whamcloud.com/job/lustre-master/3811</a> <br/> Distro/Arch: RHEL7.5/aarch64 (client), RHEL7.5/x86_64 (server) LU-11597

sanityn test 16a failed with direct I/O

Bug Major Resolved Fixed Jian Yu Jian Yu arm ppc Thu, 1 Nov 2018 17:48:32 +0000 Fri, 16 Dec 2022 14:59:54 +0000 Tue, 18 Jan 2022 14:21:29 +0000 Lustre 2.12.0 Lustre 2.13.0 Lustre 2.12.2 Lustre 2.12.3 Lustre 2.14.0 Lustre 2.12.4 Lustre 2.12.5 Lustre 2.15.0 0 4 <p>Page size of ARM processor is 64kB:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># uname -a Linux trevis-79vm36.trevis.whamcloud.com 4.14.0-49.13.1.el7a.aarch64 #1 SMP Thu Sep 27 14:45:52 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux # getconf PAGE_SIZE 65536 </pre> </div></div> <p>while it's 4kB on x86_64 processor:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># uname -a Linux trevis-58vm4.trevis.whamcloud.com 3.10.0-862.14.4.el7_lustre.x86_64 #1 SMP Fri Oct 12 14:51:33 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux # getconf PAGE_SIZE 4096 </pre> </div></div> <p>On an ARM client with x86_64 servers, writing a file less than 65536 bytes with direct I/O mode hit "-EINVAL" failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># yes | dd of=/mnt/lustre/f1 bs=65535 count=1 oflag=direct dd: error writing ‘/mnt/lustre/f1’: Invalid argument 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00356834 s, 0.0 kB/s </pre> </div></div> <p>Debug log on client showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:00000001:0.0:1541493387.509116:1776:18933:0:(cl_io.c:558:cl_io_start()) Process entered 00000080:00000001:0.0:1541493387.509118:1936:18933:0:(vvp_io.c:1036:vvp_io_write_start()) Process entered 00000080:00200000:0.0:1541493387.509121:1936:18933:0:(vvp_io.c:1061:vvp_io_write_start()) f1: write [0, 65535) 00000020:00000001:0.0:1541493387.509123:1984:18933:0:(cl_object.c:413:cl_object_maxbytes()) Process entered 00020000:00000002:0.0:1541493387.509124:2096:18933:0:(lov_object.c:1075:lov_conf_freeze()) To take share lov(ffff80002f4e0480) owner (null)/ffff8000300f4400 00020000:00000002:0.0:1541493387.509127:2096:18933:0:(lov_object.c:2092:lov_lsm_addref()) lsm ffff8000329f9a80 addref 2/0 by ffff8000300f4400. 00020000:00000002:0.0:1541493387.509129:2096:18933:0:(lov_object.c:1083:lov_conf_thaw()) To release share lov(ffff80002f4e0480) owner (null)/ffff8000300f4400 00000020:00000001:0.0:1541493387.509131:2016:18933:0:(cl_object.c:421:cl_object_maxbytes()) Process leaving (rc=17592186040320 : 17592186040320 : ffffffff000) 00000080:00000001:0.0:1541493387.509137:1968:18933:0:(vvp_io.c:1133:vvp_io_write_start()) Process leaving (rc=18446744073709551594 : -22 : ffffffffffffffea) 00000020:00000001:0.0:1541493387.509139:1808:18933:0:(cl_io.c:570:cl_io_start()) Process leaving (rc=18446744073709551594 : -22 : ffffffffffffffea) </pre> </div></div> <p>Looking into vvp_io_write_start()-&gt;__generic_file_write_iter()...</p> <p>On x86_64 client with 4kB page size, writing a file less than 4096 bytes with direct I/O mode also hit "-EINVAL" failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># yes | dd of=/mnt/lustre/f2 bs=4095 count=1 oflag=direct dd: error writing ‘/mnt/lustre/f2’: Invalid argument 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.0014026 s, 0.0 kB/s </pre> </div></div> <p>So, under direct I/O mode, the bytes to be written at a time should not be less than one page size. I'm creating a patch to update the test scripts.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33636" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33636</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11597" title="sanityn test 16a failed with direct I/O" class="issue-link" data-issue-key="LU-11597"><del>LU-11597</del></a> tests: fix O_DIRECT test usage for ARM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e26c7a9c6e747de442d5f37cf14b352ca3e4b365</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33636/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33636/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11597" title="sanityn test 16a failed with direct I/O" class="issue-link" data-issue-key="LU-11597"><del>LU-11597</del></a> tests: fix O_DIRECT test usage for ARM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f602b5ec7f45713122abd615a97a13d7c97d460e</p> <p>Landed for 2.12</p> <p>I'm reopening this ticket because sanityn test 16a, as mentioned in the description, is still failing for ARM. </p> <p>Logs for recent test failures are at<br/> <a href="https://testing.whamcloud.com/test_sets/f1121dd4-fdef-11e8-b837-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f1121dd4-fdef-11e8-b837-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/1de368ba-fa38-11e8-bb6b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1de368ba-fa38-11e8-bb6b-52540065bddc</a></p> <p>Also, PPC client testing fails this test 100% of the time; <a href="https://testing.whamcloud.com/test_sets/f25e7616-4a6e-11ea-b69a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f25e7616-4a6e-11ea-b69a-52540065bddc</a>.</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37561" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37561</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11597" title="sanityn test 16a failed with direct I/O" class="issue-link" data-issue-key="LU-11597"><del>LU-11597</del></a> tests: skip sanityn tests for PPC<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bfe402a758e752124a0d081f5bb5bde4b95566bb</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37589" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37589</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11597" title="sanityn test 16a failed with direct I/O" class="issue-link" data-issue-key="LU-11597"><del>LU-11597</del></a> test: fix sanityn 16a to align page size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c23ab3e322cfc6b585b39c97d6bbec77127e6f2b</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37561/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37561/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11597" title="sanityn test 16a failed with direct I/O" class="issue-link" data-issue-key="LU-11597"><del>LU-11597</del></a> tests: skip sanityn tests for PPC<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c27e5fe50ca3de4c9d3dbb024a0704ee3cc4e15c</p> <p>The patch that landed, <a href="https://review.whamcloud.com/37561/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37561/</a>, puts sanityn tests 16a and 71a on the ALWAYS_EXCEPT list for PPC client testing. This ticket should remain open until those tests are fixed and the tests are taken off the list. </p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40660" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40660</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11597" title="sanityn test 16a failed with direct I/O" class="issue-link" data-issue-key="LU-11597"><del>LU-11597</del></a> tests: skip sanityn tests for PPC<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 7d20d3c6abfcac939ec9c7020484882d5e8067b7</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40660/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40660/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11597" title="sanityn test 16a failed with direct I/O" class="issue-link" data-issue-key="LU-11597"><del>LU-11597</del></a> tests: skip sanityn tests for PPC<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: d0a53964f1bd7d8a21fc987df8d2f250b04f9ada</p> <p>Only sanityn test 16a fails now,  change the title.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/37589/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37589/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11597" title="sanityn test 16a failed with direct I/O" class="issue-link" data-issue-key="LU-11597"><del>LU-11597</del></a> test: Fix sanityn 16a failed on arm<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: eb704aecfaad2a6256d1e2e48cdfadbabb07e5cb</p> Duplicate Related LU-11527 LU-10929 LU-10300 Development Rank 1|i005hr: Rank (Obsolete) 9223372036854775807 Severity