https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-25 Lustre <p>We have now verified at least twice that the ldlm shrinker is sometimes getting stuck waiting for a network transaction to complete. We are of the opinion that this is an architectural problem in lustre. A shrinker should NEVER block on network transactions.</p> <p>In particular, the ldlm_cli_pool_shrink() shrinker is the one that has been giving us trouble in 1.8 for as long as a year. It was only a couple of months ago that it was finally brought to my attention, and the problem is intermittent enough that is has been difficult to catch the problem as it happens. Now both Brian and I have caught the problem with crash, and we have some other sysreq console backtraces that the sysadmins have gotten us. The problem always looks like the following, at least from get_page_from_freelist() to ptlrpc_queue_wait():</p> <p>ptlrpc_queue_wait<br/> lustre_msg_set_opc<br/> ptlrpc_at_set_req_timeout<br/> lustre_msg_buf<br/> ldlm_cli_cancel_req<br/> ldlm_cli_cancel_list<br/> ldlm_cancel_passed_policy<br/> ldlm_cancel_lru<br/> ldlm_cli_pool_shrink<br/> ldlm_cli_pool_shrink<br/> ldlm_pool_shrink<br/> ldlm_namespace_move_locked<br/> ldlm_pools_shrink<br/> ldlm_pools_cli_shrink<br/> shrink_slab<br/> get_page_from_freelist<br/> __alloc_pages<br/> alloc_pages_current<br/> tcp_sendmsg<br/> inet_sendmsg<br/> do_sock_write<br/> sock_aio_write<br/> do_sync_write<br/> sys_getsockname<br/> vfs_write<br/> sys_write</p> <p>It is not clear why we are not getting the reply to the cancel request in a reasonable time. And it appears that this thread can be stuck in ptlrpc_queue_wait() for at least an hour. We are also not yet sure why that should block for so long.</p> <p>But hopefully we can agree that a blocking operation like that should NOT happen in the shrinker. You'll notice that the kernel entry point did not even involve lustre here. We have seen other entry points, such as page faults, that also needed to call get_page_from_freelist() and then got stuck in lustre's ldlm client pool shrinker.</p> <p>Brian Behlendorf voiced concerns about this independently in <a href="https://bugzilla.lustre.org/show_bug.cgi?id=23598#c31" class="external-link" target="_blank" rel="nofollow noopener">comment #31</a> of <a href="https://bugzilla.lustre.org/show_bug.cgi?id=23598" class="external-link" target="_blank" rel="nofollow noopener">Oracle bug 23598</a>.</p> <p>We were hoping that Oleg would take a look at this. Eric might also be interested, since this appears to be an architectural flaw.</p> <p>Our immediate need is to fix this particular shrinker, but we should probably make it policy to never perform blocking network transactions in shrinkers.</p> <p>Brian suggested that a quick work-around is to just disable this shrinker, since shrinkers are "best effort". As he suggested in <a href="https://bugzilla.lustre.org/show_bug.cgi?id=23598#c31" class="external-link" target="_blank" rel="nofollow noopener">comment #31</a>, the longer term fix may be to make the shrink asynchronous, handling the blocking network transactions in another thread.</p> LU-25

Blocking network request in ldlm shrinker

Bug Critical Resolved Fixed Niu Yawei Christopher Morrone Thu, 16 Dec 2010 11:29:01 +0000 Tue, 19 Jul 2011 19:43:15 +0000 Mon, 25 Apr 2011 07:16:54 +0000 Lustre 1.8.6 Lustre 2.1.0 0 14 <p>Assigning to Oleg for analysis. </p> <p>The LDLM_ASYNC fix suggested in the Oracle bug seems reasonable. Should I go with that for now?</p> <p>See <a href="https://bugzilla.lustre.org/show_bug.cgi?id=23598" class="external-link" target="_blank" rel="nofollow noopener">BZ 23598</a> for comments as we work with this bug. </p> <p>Ok, I just went through the code, and we only really have 3 shrinkers registered in 1.8:<br/> ldlm_pools_cli_shrinker, ldlm_pools_srv_shrinker, llap_shrink_cache.</p> <p>Only ldlm_pools_cli_shrinker does blocking RPCs and this could be averted by a single replacement of LDLM_SYNC to LDLM_ASYNC in ldlm_cli_pool_shrink().<br/> The other two don't cause any RPCs.</p> <p>In 2.x in addition to the other three we had in 1.8 another shrinker is added, lu_cache_shrink. I looked through possible pathes (loo_object_delete and loo_object_free) and none of them seem to call any RPCs.<br/> So as such you probably hit the last blocking shrinker piece remaining.</p> <p>Assigned to Niu, to implement the patch.</p> <p>"But hopefully we can agree that a blocking operation like that should NOT happen in the shrinker"</p> <p>I don't see why a shrinker can't do blocking operation, I think it's totally depends on the gfp masks passed from caller, maybe we should just take care of the gfp mask correctly in shrinker. Did I miss anything?</p> <p>I suspect this is a deadlock caused by ldlm_cli_pool_shrink() doing blocking operation without checking gfd mask, canceling lock could trigger io and other network RPCs, which is deadlock prone if the caller is already in an io/networking context.</p> <p>I'll go through the code to see if deadlock will happen in such case. Of course, it'll be great if there are full stack traces on both client and server. </p> <p>&gt;&gt; I don't see why a shrinker can't do blocking operation, I think it's totally depends on the gfp masks passed from caller, maybe we should just take care of the gfp mask correctly in shrinker. Did I miss anything?</p> <p>in some cases (tcp/ip internals or vmalloc, iirc) gfp is not used properly.</p> <p>What is the exact Lustre version? I recall a fix in 1.8 that added a check for GFP mask before doing anything in the dlm shrimpers. My recollection is that LLNL is based on 1.8.2, so maybe this fix is not included in your release?</p> <p>We are seeing the issues with 1.8.4, but I am certain that 1.8.5+ will have the same problem. Brian Behlendorf and others have noted the same issue with the 2.X code during kdmu branch development as well.</p> <p>We did indeed hit at least two GFP mask issues in the past year, but I do not think that this is not one of them.</p> <p>I am not sure that you can claim that the kernel is using the "wrong" GFP mask. To make that claim, you would basically be saying that the kernel should almost never be allowed to use GFP_KERNEL.</p> <p>I do not think that __GFP_IO and __GFP_FS are a license to block indefinitely. Or even for 15 minutes. Certainly no one in user space expects to write into a TCP socket and have it block for half of an hour before sending because lustre needed to block on an RPC before a page could be allocated.</p> <p>FYI, you can see the one-liner fix that I put in our tree here:</p> <p> <a href="https://github.com/chaos/lustre/tree/1.8.4-llnl" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/chaos/lustre/tree/1.8.4-llnl</a></p> <p>It is going through two weeks of testing on hype, along with testing the latest chaos 4.4-1 point release. After that, it will being rolling out onto all of the lustre clients.</p> <p>Given the high latency and unstableness of network, I tend to agree with Christopher that sending RPC in a shrinker is unwise. </p> <p>The patch of of changing LDLM_SYNC to LDLM_ASYNC looks ok to me, well, the side defect of the patch is that the ldlm_cli_pool_shrink() will not work as kernel expected, and the possiblity of mem alloc failure is increased, however, I don't see any other better solutions so far. </p> <p>I think we can land this patch to the b1_8/master if no one object. Andreas, Oleg, any comments? </p> <p>At the very least we should wait for testing results from hyperion before this is landed.</p> <p>On our smaller hype cluster (96 nodes, 1536 processes), testing has gone even better than expected. I suspect that this fix was a big part of that. I think this should definitely go in in both 1.8 and master.</p> <p>This patch needs to be landed on master too. I pushed the patch to gerrit:</p> <p> <a href="http://review.whamcloud.com/277" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/277</a></p> <p>Integrated in reviews-patchless-centos5 #72 (See <a href="http://build.whamcloud.com/job/reviews-patchless-centos5/72/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/reviews-patchless-centos5/72/</a>)<br/> <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a>: Use LDLM_ASYNC with ldlm_cancel_lru to avoid blocking.</p> <p>Christopher J. Morrone : <a href="http://git.whamcloud.com/gitweb/?p=lustre.git;a=commit&amp;a=commit&amp;h=960557722df2194526792f581e7dadc84247e408" class="external-link" target="_blank" rel="nofollow noopener">http://git.whamcloud.com/gitweb/?p=lustre.git;a=commit&amp;a=commit&amp;h=960557722df2194526792f581e7dadc84247e408</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_pool.c</li> </ul> <p>Integrated in reviews-ubuntu #176 (See <a href="http://build.whamcloud.com/job/reviews-ubuntu/176/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/reviews-ubuntu/176/</a>)<br/> <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a>: Use LDLM_ASYNC with ldlm_cancel_lru to avoid blocking.</p> <p>Christopher J. Morrone : <a href="http://git.whamcloud.com/gitweb/?p=lustre.git;a=commit&amp;a=commit&amp;h=960557722df2194526792f581e7dadc84247e408" class="external-link" target="_blank" rel="nofollow noopener">http://git.whamcloud.com/gitweb/?p=lustre.git;a=commit&amp;a=commit&amp;h=960557722df2194526792f581e7dadc84247e408</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_pool.c</li> </ul> <p>Integrated in reviews-centos5 #339 (See <a href="http://build.whamcloud.com/job/reviews-centos5/339/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/reviews-centos5/339/</a>)<br/> <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a>: Use LDLM_ASYNC with ldlm_cancel_lru to avoid blocking.</p> <p>Christopher J. Morrone : <a href="http://git.whamcloud.com/gitweb/?p=lustre.git;a=commit&amp;a=commit&amp;h=960557722df2194526792f581e7dadc84247e408" class="external-link" target="_blank" rel="nofollow noopener">http://git.whamcloud.com/gitweb/?p=lustre.git;a=commit&amp;a=commit&amp;h=960557722df2194526792f581e7dadc84247e408</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_pool.c</li> </ul> <p>I <em>would</em> like to understand the mechanics of the hangs when LRU shrinking was doing sync lock cancels. I suspect having PF_MEMALLOC set in the thread sending requests could have allocated memory too aggressively so that replies from the server were getting dropped - for example in lnet_parse() where it calls lnet_msg_alloc(). Is it worthwhile trying to re-create this bug with better logging of allocation failures?</p> <p>Christopher, could you please let us know if you have the patch from bug 21776 applied (the one that set PF_MEMALLOC on the outgoing path)?</p> <p>No, we do not have the patch from 21776.</p> <p>I wonder if we can have some failure case logs from the client before the patch? The question is why the ptlrpc_queue_wait sits there for hours. There must be some clues in the logs. Unable to allocate memory? Retrying sending in a loop? Something else?<br/> Otherwise as Eric properly indicates we don't have total understanding why it hangs, even though we are able to treat the symptom.</p> <p>Yes, it bothers me too that we don't know why the rpc was hanging.</p> <p>But sadly no, we don't have any good logs that explain it. I am also not aware of a reliable reproducer, outside of actual production usage. And to make matters worse, it was usually on the secure network that this happened.</p> <p>I was never able to get detailed enough logs to explain why it was stuck. I was usually alerted to the problem so long after it happened that it took quite a while to even definitively determine that things were getting stuck in the shrinker.</p> <p>I think this issue should be on the 2.1-blocker list until landed.</p> <p>Hi, Chris</p> <p>The stack trace of this ticket is for 1.8, the 2.0 code is different from 1.8, in 2.0, the ldlm_cli_pool_shrink() should always pass all the cancel work to another thread then wait for the work done, so it's impossible to see the ldlm_cli_pool_shrink() calling ptlrpc_queue_wait() directly. </p> <p>I checked the bz23598, and looks it's a zfs related issue. Did you ever see such problem on 2.0 system (not the kdmu branch)? </p> <p>Thanks<br/> Niu</p> <p>Niu</p> <p>I think that LLNL are carrying an equivalent patch in their 2.1 version to avoid ever hitting this issue so probably do not have such a stack trace</p> <p>Peter</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./build_type=client,platform=el5-x86_64/24/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » client,el5-x86_64 #24</a><br/> <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a>: Use LDLM_ASYNC with ldlm_cancel_lru to avoid blocking.</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=3dcb5fb7b79949d780bc82b1439622989516e34f" class="external-link" target="_blank" rel="nofollow noopener">3dcb5fb7b79949d780bc82b1439622989516e34f</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_pool.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master-centos5/191/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master-centos5 #191</a><br/> <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a>: Use LDLM_ASYNC with ldlm_cancel_lru to avoid blocking.</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-release.git&amp;a=commit&amp;h=3dcb5fb7b79949d780bc82b1439622989516e34f" class="external-link" target="_blank" rel="nofollow noopener">3dcb5fb7b79949d780bc82b1439622989516e34f</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_pool.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./build_type=client,platform=el5-i686/24/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » client,el5-i686 #24</a><br/> <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a>: Use LDLM_ASYNC with ldlm_cancel_lru to avoid blocking.</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=3dcb5fb7b79949d780bc82b1439622989516e34f" class="external-link" target="_blank" rel="nofollow noopener">3dcb5fb7b79949d780bc82b1439622989516e34f</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_pool.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./build_type=server,platform=el6-x86_64/24/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » server,el6-x86_64 #24</a><br/> <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a>: Use LDLM_ASYNC with ldlm_cancel_lru to avoid blocking.</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=3dcb5fb7b79949d780bc82b1439622989516e34f" class="external-link" target="_blank" rel="nofollow noopener">3dcb5fb7b79949d780bc82b1439622989516e34f</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_pool.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./build_type=client,platform=ubuntu-x86_64/24/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » client,ubuntu-x86_64 #24</a><br/> <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a>: Use LDLM_ASYNC with ldlm_cancel_lru to avoid blocking.</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=3dcb5fb7b79949d780bc82b1439622989516e34f" class="external-link" target="_blank" rel="nofollow noopener">3dcb5fb7b79949d780bc82b1439622989516e34f</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_pool.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./build_type=client,platform=el6-i686/24/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » client,el6-i686 #24</a><br/> <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a>: Use LDLM_ASYNC with ldlm_cancel_lru to avoid blocking.</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=3dcb5fb7b79949d780bc82b1439622989516e34f" class="external-link" target="_blank" rel="nofollow noopener">3dcb5fb7b79949d780bc82b1439622989516e34f</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_pool.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./build_type=server,platform=el5-x86_64/24/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » server,el5-x86_64 #24</a><br/> <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a>: Use LDLM_ASYNC with ldlm_cancel_lru to avoid blocking.</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=3dcb5fb7b79949d780bc82b1439622989516e34f" class="external-link" target="_blank" rel="nofollow noopener">3dcb5fb7b79949d780bc82b1439622989516e34f</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_pool.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./build_type=client,platform=el6-x86_64/24/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » client,el6-x86_64 #24</a><br/> <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a>: Use LDLM_ASYNC with ldlm_cancel_lru to avoid blocking.</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=3dcb5fb7b79949d780bc82b1439622989516e34f" class="external-link" target="_blank" rel="nofollow noopener">3dcb5fb7b79949d780bc82b1439622989516e34f</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_pool.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./build_type=server,platform=el5-i686/24/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » server,el5-i686 #24</a><br/> <a href="https://jira.whamcloud.com/browse/LU-25" title="Blocking network request in ldlm shrinker" class="issue-link" data-issue-key="LU-25"><del>LU-25</del></a>: Use LDLM_ASYNC with ldlm_cancel_lru to avoid blocking.</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=3dcb5fb7b79949d780bc82b1439622989516e34f" class="external-link" target="_blank" rel="nofollow noopener">3dcb5fb7b79949d780bc82b1439622989516e34f</a><br/> Files : </p> <ul> <li>lustre/ldlm/ldlm_pool.c</li> </ul> <p>Landed for 2.1</p> <p>I see that the use of LDLM_ASYNC wasn't included in 1.8.6-wc1. Since it was in LLNL's chaos 1.8.4, then should it be landed to b1_8 now?</p> Development Rank 1|hzv9if: Rank (Obsolete) 5075 Severity