https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1431 Lustre <p>Currently Lustre maximum buffer size for a RPC sending I/O is 1MB. This work looks to change the amount of data transfer to allow the data sent to be a size to achieve peak performance with large I/O transfers to the back end disk. Also an additional benefit is the reduction in the round trip time to send the data.</p> LU-1431

Support for larger than 1MB sequential I/O RPCs

New Feature Major Resolved Fixed Peter Jones James A Simmons performance Tue, 22 May 2012 11:55:10 +0000 Tue, 8 Dec 2020 05:50:17 +0000 Tue, 14 May 2013 15:34:52 +0000 Lustre 2.4.0 Lustre 2.4.0 0 19 <p>Initial patch at <a href="http://review.whamcloud.com/#change,2872" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2872</a></p> <p>Thanks James. I have asked Sergii to keep this bug updated with our progress.</p> <p>Nathan, Shadow,<br/> any update on this ticket? The last time we spoke about this, the patch from James wasn't working properly due to bad interaction with the 1MB bulk readdir RPCs on the MDS, and some other issues.</p> <p>If you have a patch that allows larger than 1MB bulk RPCs, it would be great to refresh <a href="http://review.whamcloud.com/2872" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2872</a> with a working patch. In order to avoid potential issues with performance decrease under some workloads, it might make sense to land the patch initially with the default maximum bulk RPC size still at 1MB. This will allow users to test with large RPCs (load test and performance) and provide feedback, with minimal risk. Alternately, if you have some performance metrics that compare 1MB/4MB performance under different loads (FPP, SSF, single client, many clients) that would be great.</p> <p>Xyratex <a href="http://jira-nss.xy01.xyratex.com:8080/browse/MRP-319" class="external-link" target="_blank" rel="nofollow noopener">MRP-319</a></p> <p>This should be finished up in the next two weeks.</p> <p>We split it into a second cleanup patch:<br/> MRP-687 PTLRPC_BRW_MAX_SIZE usage cleanup.</p> <p> Clean-up the layering in current code by eliminating direct usage of<br/> PTLRPC_BRW_MAX_SIZE macro outside of the ptlrpc module. This should help<br/> us acheive "floating" max brw size value across the cluster, which in<br/> turn should help with 4MB IO task.</p> <p>Nathan, thanks for the update.</p> <p>Splitting the patch up definitely makes sense, since I recall there are a number of places that don't differentiate between PTLRPC_BRW_MAX_SIZE and cl_max_pages_per_rpc properly.</p> <p>I assume you know the MRP-nnn URLs don't work outside of the Xyratex intranet, which is fine as long as the LU ticket gets updated with relevant information when the patches are submitted.</p> <p>Andreas,</p> <p>For the second patch (the actual one that changes BRW size to 4MB) we need a new connect flag OBD_CONNECT_MULTIBULK. Comment in lustre_idl.h suggests that such changes must be approved by senior engineers before even sending the patch that reserves it for future use. So, I'm asking you for the approval and will push review request right after it. Thanks.</p> <p>Sergii, could you please explain more fully why the new connect flag is needed. I thought there was already an existing OBD_CONNECT_BRW_SIZE feature which allows the client and server to negotiate the maximum BRW RPC size already...</p> <p>As for requesting a flag assignment, this is to avoid conflicting users of OBD_CONNECT flags, which would render the feature bits useless.</p> <p>Will this patch be submitted to Gerrit soon for inspection and testing?</p> <p>Could you please update this bug with the current status of the patch. </p> <p>We have finished internal inspection - I will ask deen to upload last version in Monday.</p> <p><a href="http://review.whamcloud.com/#change,4876" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4876</a></p> <p>PTLRPC_MAX_BRW_SIZE cleanup: <a href="http://review.whamcloud.com/#change,4876" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4876</a></p> <p>1. Instead of using one PTLRPC_MAX_BRW_SIZE all over the place, introduce MD_MAX_BRW_SIZE, OSC_MAX_BRW_SIZE, FILTER_MAX_BRW_SIZE (should it be OFD_MAX_BRW_SIZE now?) and auxiliary ONE_MB_BRW_SIZE.<br/> ptlrpc still uses its own PTLRPC_MAX_BRW_SIZE, while other subsystems now use their corresponding macros. The actual 4MB IO patch will change only PTLRPC_MAX_BRW_SIZE and OSC_MAX_BRW_SIZE to 4MB, leaving other subsystems intact.</p> <p>2. From the original 4MB IO patch (bz16900), take in code which embeds obd_connect_data into obd_export in order to store the other node's ocd_brw_size value. The idea is to have a "floating" brw size across the cluster: during connection, nodes need to decide on the suitable brw size for both of them, which is min(node1_brw_size, node2_brw_size).</p> <p>Updated patch has been pushed to Gerrit.</p> <p>The actual 4MB I/O patch: <a href="http://review.whamcloud.com/#change,4993" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4993</a></p> <p>As part of the task, I've developed a couple of tests. The first one, test_231a, checks that in case of a single 4MB transfer only one BRW RPC is sent. It depends on the information from /proc/fs/lustre/obdfilter/$OST/brw_stats and it seems that there are some issues with this statiscics in your codebase, because despite the fact of the I/O the file is empty (only rows/columns names are shown). I haven't looked into this yet, so maybe you will have some ideas. Thanks.</p> <p>Andreas, thank you for the hint regarding osd-ldiskfs. I've fixed the test and have just pushed an updated patch to the Gerrit. It works on my local setup, so I think there will be no problems.<br/> So, the current state of the task is that both patches successfully pass all the tests (Well, to be 100% sure we need to wait for the test results for the latest push, but I'm pretty sure that there will be no issues). </p> <p>I see that these patches fail testing at random places that never failed before, so this is worrysome.<br/> What sort of testing did these patches see on your end I wonder?</p> <p>I don't think that these patches fail at random places completely. The latest test runs for both patches fail in common places: replay-single and sanity-quota.</p> <p>As for the testing, the thing is that both patches differ from our original ones due to requests from Andreas, so I don't think that our testing results for the original patches are relevant.</p> <p>Sergii, please see if you can reproduce the failures; it's our responsibility to fix any problems here.</p> <p>I noticed that the llite readahead window increment (RAS_INCREASE_STEP) was also based on PTLRPC_MAX_BRW_SIZE, but this is too large for PTLRPC_MAX_BRW_SIZE of 32MB when the actual cl_max_pages_per_rpc is only 1MB. Instead, limit the readahead window growth to match the inode-&gt;i_blkbits (current default min(PTLRPC_MAX_BRW_SIZE * 2, 4MB)), which is still reasonable regardless of the blocksize. This also allows tuning the readahead on a per-inode basis in the future, depending on which OSTs the file is striped over, by fixing the i_blkbits value.</p> <p><a href="http://review.whamcloud.com/5230" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5230</a></p> <p>Deen, sorry about the problems with the 32MB RPC size. That would have allowed us much better flexibility for testing and updated hardware in the future. As stated in the patch, there are a number of issues that still need to be addressed:</p> <blockquote> <p>Sorry, it seems that the 32MB RPC size is too large to handle with the current code. Sorry for the confusion. It makes sense to revert PTLRPC_MAX_BRW_SIZE to 4MB for this patch, and we can resolve the problems with larger RPC size in follow-on patches.</p> <p>There are a number of problems found at 32MB that can be fixed independently:</p> <p> osd_thread_info.osd_iobuf.dr_blocks[] is 512kB<br/> osd_thread_info.osd_iobuf.dr_pages[] is 64kB<br/> osd_thread_info.oti_created[] is 32kB and is unused and can be removed<br/> oti_thread_info uses OBD_ALLOC() instead of OBD_ALLOC_LARGE()<br/> all OST RPC threads allocate the same large OST_MAXREQSIZE buffers, but this is only needed for the OST_IO_PORTAL<br/> osd_thread_info.osd_iobuf is only needed for OST_IO_PORTAL and does not need to be allocated for other threads<br/> with the larger RPC buffers, there should be fewer total buffers allocated, see comments in <a href="http://review.whamcloud.com/4940" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4940</a></p></blockquote> <p>The current test result for this patch shows good improvement on FPP write, but a net loss for other IO loads.<br/> Do you have any IO performance data that confirms or contradicts the below results?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>IOR Single-shared file Date RPC size clients write read 2013/02/03 4MB 105 7153 8200 2013/02/03 1mb 105 7996 9269 IOR File-per-process Date RPC size clients write read 2013/02/03 4MB 105 9283 6000 2013/02/03 1mb 106 7233 6115 </pre> </div></div> <p>If this is the case, we could still e.g. default to sending 4MB write RPCs if there is enough data in cache and the client holds an exclusive DLM lock.</p> <p><a href="https://jira.whamcloud.com/browse/LUDOC-80" title="4MB RPC Doc Changes" class="issue-link" data-issue-key="LUDOC-80"><del>LUDOC-80</del></a> landed, closing bug.</p> <p>Our testing system shows, that there is failed test eplay-ost-single.test_5</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Lustre: DEBUG MARKER: == replay-ost-single test 5: Fail OST during iozone == 21:21:13 (1369851673) Lustre: Failing over lustre-OST0000 LustreError: 11-0: an error occurred <span class="code-keyword">while</span> communicating with 0@lo. The ost_write operation failed with -19 LustreError: Skipped 1 previous similar message Lustre: lustre-OST0000-osc-ffff8800514d3400: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: shutting down <span class="code-keyword">for</span> failover; client state will be preserved. Lustre: OST lustre-OST0000 has stopped. Lustre: server umount lustre-OST0000 complete LustreError: 137-5: UUID <span class="code-quote">'lustre-OST0000_UUID'</span> is not available <span class="code-keyword">for</span> connect (no target) LustreError: Skipped 1 previous similar message LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: Lustre: 16962:0:(ldlm_lib.c:2195:target_recovery_init()) RECOVERY: service lustre-OST0000, 2 recoverable clients, last_transno 1322 Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/loop1 with recovery enabled Lustre: 2398:0:(ldlm_lib.c:1021:target_handle_connect()) lustre-OST0000: connection from lustre-MDT0000-mdtlov_UUID@0@lo recovering/t0 exp ffff88005ca19c00 cur 1369851700 last 1369851697 Lustre: 2398:0:(ldlm_lib.c:1021:target_handle_connect()) Skipped 3 previous similar messages Lustre: lustre-OST0000: Will be in recovery <span class="code-keyword">for</span> at least 1:00, or until 2 clients reconnect Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to lustre-OST0000 (at 0@lo) Lustre: Skipped 1 previous similar message LustreError: 1716:0:(osc_request.c:1232:check_write_rcs()) Unexpected # bytes transferred: 65536 (requested 32768) LustreError: 1716:0:(osc_request.c:1232:check_write_rcs()) Unexpected # bytes transferred: 2097152 (requested 1048576) Lustre: lustre-OST0000: received MDS connection from 0@lo Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0000_UUID now active, resetting orphans Lustre: DEBUG MARKER: iozone rc=1 Lustre: DEBUG MARKER: replay-ost-single test_5: @@@@@@ FAIL: iozone failed </pre> </div></div> <p>This messages looks related to 4mb IO patch</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 1716:0:(osc_request.c:1232:check_write_rcs()) Unexpected # bytes transferred: 65536 (requested 32768) LustreError: 1716:0:(osc_request.c:1232:check_write_rcs()) Unexpected # bytes transferred: 2097152 (requested 1048576) </pre> </div></div> <p>I believe, that this test is failed in intel's master branch, but they skip it as SLOW during testing<br/> <a href="https://maloo.whamcloud.com/test_sets/dd033a98-7264-11e2-aad1-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/dd033a98-7264-11e2-aad1-52540035b04c</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">test_5 SKIP 0 0 skipping SLOW test 5 </pre> </div></div> <p>Could you, please, start this test (it marked as SLOW) and check if it failed?</p> <p>Artem</p> <p>Could you please open a new ticket for this failure so we can track it?</p> <p>Thanks</p> <p>Peter</p> <p><a href="https://jira.whamcloud.com/browse/LU-3438" title="replay-ost-single test_5 failed with error int check_write_rcs() &quot;Unexpected # bytes transferred&quot;" class="issue-link" data-issue-key="LU-3438"><del>LU-3438</del></a> is created.</p> <p>Thanks Artem!</p> <p>Xyratex-bug-id: <a href="http://jira-nss.xy01.xyratex.com:8080/browse/MRP-687" class="external-link" target="_blank" rel="nofollow noopener">MRP-687 </a><br/> Xyratex-bug-id: <a href="http://jira-nss.xy01.xyratex.com:8080/browse/MRP-319" class="external-link" target="_blank" rel="nofollow noopener">MRP-319 </a></p> Related LUDOC-80 LU-2424 LU-4533 LU-3308 LU-2791 LU-2816 LU-2598 LU-2790 LU-3438 LU-2748 LUDOC-80 LU-2702 LU-2756 LU-2748 Bugzilla ID 16900.0 Development Epic lnet Rank 1|hzv3br: Rank (Obsolete) 4038