https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12609 Lustre <p>It is important to limit the total number of fragments allowed in an RPC to avoid issues with the on disk file system which can occur with too many fragments.</p> <p>For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>May 27 03:21:02 oss2c104 kernel: Lustre: 5711:0:(osd_handler.c:1155:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 May 27 03:21:02 oss2c104 kernel: Lustre: 5711:0:(osd_handler.c:1155:osd_trans_dump_creds()) Skipped 4 previous similar messages May 27 03:21:02 oss2c104 kernel: Lustre: 5711:0:(osd_handler.c:1162:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 1/1/0 May 27 03:21:02 oss2c104 kernel: Lustre: 5711:0:(osd_handler.c:1162:osd_trans_dump_creds()) Skipped 4 previous similar messages May 27 03:21:02 oss2c104 kernel: Lustre: 5711:0:(osd_handler.c:1172:osd_trans_dump_creds()) write: 2/17434/0, punch: 0/0/0, quota 5/5/0 May 27 03:21:02 oss2c104 kernel: Lustre: 5711:0:(osd_handler.c:1172:osd_trans_dump_creds()) Skipped 4 previous similar messages May 27 03:21:02 oss2c104 kernel: Lustre: 5711:0:(osd_handler.c:1179:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 May 27 03:21:02 oss2c104 kernel: Lustre: 5711:0:(osd_handler.c:1179:osd_trans_dump_creds()) Skipped 4 previous similar messages May 27 03:21:02 oss2c104 kernel: Lustre: 5711:0:(osd_handler.c:1186:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 May 27 03:21:02 oss2c104 kernel: Lustre: 5711:0:(osd_handler.c:1186:osd_trans_dump_creds()) Skipped 4 previous similar messages May 27 03:21:02 oss2c104 kernel: LustreError: 5711:0:(osd_internal.h:1117:osd_trans_exec_op()) share1-OST004c-osd: op = 7, rb = 7 May 27 03:21:02 oss2c104 kernel: LustreError: 5711:0:(osd_internal.h:1117:osd_trans_exec_op()) Skipped 3 previous similar messages May 27 03:21:02 oss2c104 kernel: Pid: 5856, comm: ll_ost_io01_044 May 27 03:21:02 oss2c104 kernel: \x0aCall Trace: May 27 03:21:02 oss2c104 kernel: [&lt;ffffffffc074680e&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] May 27 03:21:02 oss2c104 kernel: [&lt;ffffffffc0746846&gt;] libcfs_debug_dumpstack+0x26/0x30 [libcfs] May 27 03:21:02 oss2c104 kernel: [&lt;ffffffffc0e3fcfd&gt;] osd_trans_start+0x36d/0x390 [osd_ldiskfs] May 27 03:21:02 oss2c104 kernel: [&lt;ffffffffc0f964fe&gt;] ofd_trans_start+0x6e/0xf0 [ofd] May 27 03:21:02 oss2c104 kernel: [&lt;ffffffffc0f9cc63&gt;] ofd_commitrw_write+0x943/0x1d90 [ofd] May 27 03:21:02 oss2c104 kernel: [&lt;ffffffffc0fa100b&gt;] ofd_commitrw+0x54b/0xb70 [ofd] May 27 03:21:02 oss2c104 kernel: [&lt;ffffffffc0bd39d6&gt;] obd_commitrw.constprop.39+0x2fe/0x341 [ptlrpc] May 27 03:21:02 oss2c104 kernel: [&lt;ffffffffc0bc02e0&gt;] tgt_brw_write+0xd60/0x16d0 [ptlrpc] May 27 03:21:02 oss2c104 kernel: [&lt;ffffffff810d11c4&gt;] ? update_curr+0x104/0x190 May 27 03:21:02 oss2c104 kernel: [&lt;ffffffff810cdbce&gt;] ? account_entity_dequeue+0xae/0xd0 May 27 03:21:02 oss2c104 kernel: [&lt;ffffffffc0b0a650&gt;] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] May 27 03:21:02 oss2c104 kernel: [&lt;ffffffffc0bbc460&gt;] tgt_request_handle+0x900/0x11f0 [ptlrpc] May 27 03:21:02 oss2c104 kernel: [&lt;ffffffffc0b5edc0&gt;] ptlrpc_server_handle_request+0x220/0xa90 [ptlrpc] May 27 03:21:02 oss2c104 kernel: [&lt;ffffffffc0758818&gt;] ? lc_watchdog_touch+0x68/0x180 [libcfs] May 27 03:21:02 oss2c104 kernel: [&lt;ffffffffc0b5b938&gt;] ? ptlrpc_wait_event+0x98/0x330 [ptlrpc] May 27 03:21:02 oss2c104 kernel: [&lt;ffffffff810bdc4b&gt;] ? __wake_up_common+0x5b/0x90 May 27 03:21:02 oss2c104 kernel: [&lt;ffffffffc0b6278f&gt;] ptlrpc_main+0xc3f/0x1fa0 [ptlrpc] May 27 03:21:02 oss2c104 kernel: [&lt;ffffffffc0b61b50&gt;] ? ptlrpc_main+0x0/0x1fa0 [ptlrpc] May 27 03:21:02 oss2c104 kernel: [&lt;ffffffff810b4031&gt;] kthread+0xd1/0xe0 May 27 03:21:02 oss2c104 kernel: [&lt;ffffffff810b3f60&gt;] ? kthread+0x0/0xe0 May 27 03:21:02 oss2c104 kernel: [&lt;ffffffff816c1577&gt;] ret_from_fork+0x77/0xb0 May 27 03:21:02 oss2c104 kernel: [&lt;ffffffff810b3f60&gt;] ? kthread+0x0/0xe0 </pre> </div></div> LU-12609

Limit maximum number of journal credits for an RPC

Bug Minor Open Unresolved Alex Zhuravlev Patrick Farrell Mon, 29 Jul 2019 18:33:31 +0000 Tue, 29 Oct 2019 22:11:31 +0000 0 3 <p><a href="https://review.whamcloud.com/#/c/35609/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/35609/</a></p> <p>Patrick, based on your recent comments in the patch, you indicate that the number of fragments per RPC is already being limited? Do you have some empirical evidence to suggest this is the case (eg. logs with 4MB RPCs showing random writes sending &lt;= 256 niobufs?</p> <p>Another area of investigation is to see whether the journal credits can be "slimmed down" in such cases. It is entirely possible that we are over-allocating buffer credits for such cases (eg. superblock, inode, higher-level index blocks, etc x256) when they cannot possibly be needed, even if each block was allocated to a separate group. </p> <p>Well, I noticed when writing my code I was successfully limiting the number of fragments/niobufs, but then I realized what I had done was identical to the existing "limit number of extents" code, basically.  Since every extent is one fragment, guaranteed.  (I did do some poking to check this.)</p> <p>I could do some more testing if you'd like, but I'm pretty sure.  (I didn't do random i/o, I just did strided unaligned i/o from one process with big gaps.  So each write generated an extent.)</p> <p>re: Credits, unfortunately, I am totally out of my depth in that area.</p> <p>Alex, it may be that this is avoided by the writeback cache patch, but allocating too many credits for random IO is possibly still a performance overhead that isn't needed? Not a huge priority, but maybe worth a few minutes if there is something obvious to fix. </p> Development Rank 1|i00kcv: Rank (Obsolete) 9223372036854775807 Severity