https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2432 Lustre <p>vmalloc based allocations can potentially take a very long time to complete due to a regression in the kernel. As a result, I've seen our MDS "lock up" for certain periods of time while all of the cores spin on the vmap_area_lock down in ptlrpc_alloc_rqbd.</p> <p>For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 2012-11-01 11:34:28 Pid: 34505, comm: mdt02_051 2012-11-01 11:34:28 2012-11-01 11:34:28 Call Trace: 2012-11-01 11:34:28 [&lt;ffffffff81273155&gt;] ? rb_insert_color+0x125/0x160 2012-11-01 11:34:28 [&lt;ffffffff81149f1f&gt;] ? __vmalloc_area_node+0x5f/0x190 2012-11-01 11:34:28 [&lt;ffffffff810609ea&gt;] __cond_resched+0x2a/0x40 2012-11-01 11:34:28 [&lt;ffffffff814efa60&gt;] _cond_resched+0x30/0x40 2012-11-01 11:34:28 [&lt;ffffffff8115fa88&gt;] kmem_cache_alloc_node_notrace+0xa8/0x130 2012-11-01 11:34:28 [&lt;ffffffff8115fc8b&gt;] __kmalloc_node+0x7b/0x100 2012-11-01 11:34:28 [&lt;ffffffffa05a2a40&gt;] ? cfs_cpt_vmalloc+0x20/0x30 [libcfs] 2012-11-01 11:34:28 [&lt;ffffffff81149f1f&gt;] __vmalloc_area_node+0x5f/0x190 2012-11-01 11:34:28 [&lt;ffffffffa05a2a40&gt;] ? cfs_cpt_vmalloc+0x20/0x30 [libcfs] 2012-11-01 11:34:28 [&lt;ffffffff81149eb2&gt;] __vmalloc_node+0xa2/0xb0 2012-11-01 11:34:28 [&lt;ffffffffa05a2a40&gt;] ? cfs_cpt_vmalloc+0x20/0x30 [libcfs] 2012-11-01 11:34:28 [&lt;ffffffff8114a199&gt;] vmalloc_node+0x29/0x30 2012-11-01 11:34:28 [&lt;ffffffffa05a2a40&gt;] cfs_cpt_vmalloc+0x20/0x30 [libcfs] 2012-11-01 11:34:28 [&lt;ffffffffa0922ffe&gt;] ptlrpc_alloc_rqbd+0x13e/0x690 [ptlrpc] 2012-11-01 11:34:28 [&lt;ffffffffa09235b5&gt;] ptlrpc_grow_req_bufs+0x65/0x1b0 [ptlrpc] 2012-11-01 11:34:28 [&lt;ffffffffa0927fbd&gt;] ptlrpc_main+0xd0d/0x19f0 [ptlrpc] 2012-11-01 11:34:28 [&lt;ffffffffa09272b0&gt;] ? ptlrpc_main+0x0/0x19f0 [ptlrpc] 2012-11-01 11:34:28 [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 2012-11-01 11:34:28 [&lt;ffffffffa09272b0&gt;] ? ptlrpc_main+0x0/0x19f0 [ptlrpc] 2012-11-01 11:34:28 [&lt;ffffffffa09272b0&gt;] ? ptlrpc_main+0x0/0x19f0 [ptlrpc] 2012-11-01 11:34:28 [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Here's a couple links regarding the kernel regression:</p> <ul> <li><a href="http://lkml.indiana.edu/hypermail/linux/kernel/1006.3/00091.html" class="external-link" target="_blank" rel="nofollow noopener">http://lkml.indiana.edu/hypermail/linux/kernel/1006.3/00091.html</a></li> <li><a href="https://github.com/torvalds/linux/commit/89699605fe7cfd8611900346f61cb6cbf179b10a" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/torvalds/linux/commit/89699605fe7cfd8611900346f61cb6cbf179b10a</a></li> </ul> LU-2432

ptlrpc_alloc_rqbd spinning on vmap_area_lock on MDS

Bug Minor Resolved Fixed Zhenyu Xu Prakash Surya sequoia Wed, 5 Dec 2012 18:08:28 +0000 Mon, 18 Mar 2013 09:25:06 +0000 Mon, 18 Mar 2013 09:25:06 +0000 Lustre 2.4.0 Lustre 2.4.0 0 9 <p>See: <a href="http://review.whamcloud.com/4439" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4439</a></p> <p>Thanks Prakash!</p> <p>Bobijam could you please review this patch?</p> <p>svc-&gt;srv_buf_size can be MDS_BUFSIZE = (362 + LOV_MAX_STRIPE_COUNT * 56 + 1024) ~= 110KB for MDS service, could it be problematic?</p> <p>We discussed at LAD that one problem with the request buffers is that the incoming LNET buffers (sorry, I don't have the correct LNET terms here) are allocated only large enough for the largest single request, though most requests are smaller than this. Unfortunately, as soon as a single RPC is waiting in the incoming buffer, there is no longer enough space in the buffer to receive a maximum-sized incoming request. This means that each buffer is only ever used for a single message, regardless of how many might fit.</p> <p>A solution that was discussed was to make the request buffer be 2x as large as the maximum request size and/or rounded up to the next power-of-two boundary. That would at least increase the buffer utilization to 50%, and would likely allow tens of requests per LNET buffer.</p> <p>It may be that the patch for <a href="https://jira.whamcloud.com/browse/LU-2424" title="add memory limits for ptlrpc service" class="issue-link" data-issue-key="LU-2424"><del>LU-2424</del></a> will already address this issue?</p> <p>I wasn't at LAD, so I'm unaware of that discussion. But, what trade offs are being made between the number of buffers used and the size of each? i.e why can't we just have one huge buffer, increasing the utilization to <tt>(BUFFER_SIZE-REQUEST_SIZE)/BUFFER_SIZE</tt> percent (trending towards 100% as <tt>BUFFER_SIZE</tt> grows large)? Granted I don't understand the LNET code well, so I must be missing something which makes that obviously the wrong thing to do.</p> <p>My (imperfect) understanding is that the receive buffers cannot be re-used until all of the requests therein are processed. That means the buffered are filled from the start, processed, and then returned to the incoming buffer list. If the buffer is too large, then requests sitting in the buffer may wait too long to be processed, or the buffer <em>still</em> will not be fully utilized if there is an upper limit for how long a request will wait. </p> <p>I didn't realize that we still don't have the "big request buffer" fix, then this should be the right way to fix this problem and <a href="https://jira.whamcloud.com/browse/LU-2424" title="add memory limits for ptlrpc service" class="issue-link" data-issue-key="LU-2424"><del>LU-2424</del></a>.<br/> I would suggest to have 512K or 1M as request buffer size, as Andreas said, a very large request buffer can't be reused if any of those (thousands or more) requests is pending on something, so it might have some other issues.<br/> And I still think it's a nice improvement if we only allow one thread (per CPT) to enter allocating path.</p> <p>I posted a patch for this: <br/> <a href="http://review.whamcloud.com/#change,4939" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4939</a><br/> and another patch to resolve buffer utilizaiton issue: <br/> <a href="http://review.whamcloud.com/#change,4940" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4940</a></p> <p>Liang, Why would limiting the vmalloc calls to a single thread fix the issue? That one thread will still be affected by the regression. Will the other threads still be able to service requests despite needing more request buffers? Or will they all have wait for this single thread to finish the allocations?</p> <p>I think we might not care one thread (or very few threads) spinning, because each service has tens or even hundreds of threads, and servers normally have many CPU cores, all other threads can serve requests, they will not wait for buffer allocating at all. <br/> The key issue of this ticket is vmalloc can't be parallelized, so it's a waste if all threads/CPUs try to allocate buffers at the same time. </p> <blockquote> <p>all other threads can serve requests, they will not wait for buffer allocating at all</p></blockquote> <p>Perfect, that's what I wanted to verify with you. Thanks for the clarification!</p> <p>Both <a href="http://review.whamcloud.com/4939" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4939</a> and <a href="http://review.whamcloud.com/4940" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4940</a> have landed, so I think this bug could be closed. There should only be a single thread calling vmalloc() now.</p> Related LU-2708 LU-2424 Development Rank 1|hzvdbr: Rank (Obsolete) 5764 Severity