https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-179 Lustre <p>A customer is seeing a problem on a client where the client loses access to Lustre when the node is subjected to memory pressure from an errant application.</p> <p>Lustre starts reporting -113 (No route to host) errors for certain NIDS in the filesystem despite the TCP/IP network being functional. After the memory pressure is relieved the Lustre errors remain. I am collecting logs currently.</p> <p>From the customer report:</p> <p>Lnet is reporting no-route-to-host for a significant number of OSS / MDSs (client log attached).</p> <p>Mar 29 09:23:27 cgp-bigmem kernel: <span class="error">&#91;589295.826095&#93;</span> LustreError: 4980:0:(events.c:66:request_out_callback()) @@@ type 4, status <del>113 req@ffff881d2e995400 x1363985318437337/t0 o8</del>&gt;lus03-OST0000_UUID@172.17.128.130@tcp:28/4 lens 368/584 e 0 to 1 dl 1301387122 ref 2 fl Rpc:N/0/0 rc 0/0</p> <p>but from user-space on the client, all those nodes are pingable:</p> <p>cgp-bigmem:/var/log# ping 172.17.128.130<br/> PING 172.17.128.130 (172.17.128.130) 56(84) bytes of data.<br/> 64 bytes from 172.17.128.130: icmp_seq=1 ttl=62 time=0.102 ms<br/> 64 bytes from 172.17.128.130: icmp_seq=2 ttl=62 time=0.091 ms<br/> 64 bytes from 172.17.128.130: icmp_seq=3 ttl=62 time=0.091 ms<br/> 64 bytes from 172.17.128.130: icmp_seq=4 ttl=62 time=0.090 ms</p> <p>however a lnet ping hangs:<br/> cgp-bigmem:~# lctl ping 172.17.128.130@tcp</p> <p>From another client, the ping works as expected</p> <p>farm2-head1:# lctl ping 172.17.128.130@tcp<br/> 12345-0@lo<br/> 12345-172.17.128.130@tcp</p> <p>cgp-bigmem:~# lfs check servers | grep -v active<br/> error: check 'lus01-OST0007-osc-ffff88205bd52000' Resource temporarily unavailable<br/> error: check 'lus01-OST0008-osc-ffff88205bd52000' Resource temporarily unavailable<br/> error: check 'lus01-OST0009-osc-ffff88205bd52000' Resource temporarily unavailable<br/> error: check 'lus01-OST000a-osc-ffff88205bd52000' Resource temporarily unavailable<br/> error: check 'lus01-OST000b-osc-ffff88205bd52000' Resource temporarily unavailable<br/> error: check 'lus01-OST000c-osc-ffff88205bd52000' Resource temporarily unavailable<br/> error: check 'lus01-OST000d-osc-ffff88205bd52000' Resource temporarily unavailable<br/> error: check 'lus01-OST000e-osc-ffff88205bd52000' Resource temporarily unavailable<br/> error: check 'lus02-MDT0000-mdc-ffff8880735ea000' Resource temporarily unavailable<br/> error: check 'lus03-OST0000-osc-ffff8840730a1400' Resource temporarily unavailable<br/> error: check 'lus03-OST0001-osc-ffff8840730a1400' Resource temporarily unavailable<br/> error: check 'lus03-OST0002-osc-ffff8840730a1400' Resource temporarily unavailable<br/> error: check 'lus03-OST0003-osc-ffff8840730a1400' Resource temporarily unavailable<br/> error: check 'lus03-OST0004-osc-ffff8840730a1400' Resource temporarily unavailable<br/> error: check 'lus03-OST0005-osc-ffff8840730a1400' Resource temporarily unavailable<br/> error: check 'lus03-OST0006-osc-ffff8840730a1400' Resource temporarily unavailable<br/> error: check 'lus03-OST0007-osc-ffff8840730a1400' Resource temporarily unavailable<br/> error: check 'lus03-OST0008-osc-ffff8840730a1400' Resource temporarily unavailable<br/> error: check 'lus03-OST0009-osc-ffff8840730a1400' Resource temporarily unavailable<br/> error: check 'lus03-OST000a-osc-ffff8840730a1400' Resource temporarily unavailable<br/> error: check 'lus03-OST000b-osc-ffff8840730a1400' Resource temporarily unavailable<br/> error: check 'lus03-OST000c-osc-ffff8840730a1400' Resource temporarily unavailable<br/> error: check 'lus03-OST0019-osc-ffff8840730a1400' Resource temporarily unavailable<br/> error: check 'lus03-OST001a-osc-ffff8840730a1400' Resource temporarily unavailable<br/> error: check 'lus05-OST0010-osc-ffff886070dab800' Resource temporarily unavailable<br/> error: check 'lus05-OST0012-osc-ffff886070dab800' Resource temporarily unavailable<br/> error: check 'lus05-OST0014-osc-ffff886070dab800' Resource temporarily unavailable<br/> error: check 'lus05-OST0016-osc-ffff886070dab800' Resource temporarily unavailable<br/> error: check 'lus05-OST0018-osc-ffff886070dab800' Resource temporarily unavailable<br/> error: check 'lus05-OST001a-osc-ffff886070dab800' Resource temporarily unavailable<br/> error: check 'lus05-OST001c-osc-ffff886070dab800' Resource temporarily unavailable<br/> error: check 'lus05-OST000f-osc-ffff886070dab800' Resource temporarily unavailable<br/> error: check 'lus05-OST0011-osc-ffff886070dab800' Resource temporarily unavailable<br/> error: check 'lus05-OST0013-osc-ffff886070dab800' Resource temporarily unavailable<br/> error: check 'lus05-OST0015-osc-ffff886070dab800' Resource temporarily unavailable<br/> error: check 'lus05-OST0017-osc-ffff886070dab800' Resource temporarily unavailable<br/> error: check 'lus05-OST0019-osc-ffff886070dab800' Resource temporarily unavailable<br/> error: check 'lus05-OST001b-osc-ffff886070dab800' Resource temporarily unavailable<br/> error: check 'lus05-OST001d-osc-ffff886070dab800' Resource temporarily unavailable<br/> error: check 'lus04-OST0001-osc-ffff88806e9d8c00' Resource temporarily unavailable<br/> error: check 'lus04-OST0003-osc-ffff88806e9d8c00' Resource temporarily unavailable<br/> error: check 'lus04-OST0005-osc-ffff88806e9d8c00' Resource temporarily unavailable<br/> error: check 'lus04-OST0007-osc-ffff88806e9d8c00' Resource temporarily unavailable<br/> error: check 'lus04-OST0009-osc-ffff88806e9d8c00' Resource temporarily unavailable<br/> error: check 'lus04-OST000b-osc-ffff88806e9d8c00' Resource temporarily unavailable<br/> error: check 'lus04-OST000d-osc-ffff88806e9d8c00' Resource temporarily unavailable</p> Client is running 2.6.27.45-lustre-1.8.3.ddn3.3. Connectivity is 10GigE LU-179

lustre client lockup when under memory pressure

Bug Minor Resolved Fixed Zhenyu Xu Shuichi Ihara Wed, 30 Mar 2011 04:19:43 +0000 Fri, 26 Feb 2016 15:10:54 +0000 Wed, 6 Jul 2011 21:37:58 +0000 Lustre 1.8.6 0 3 <p>Bobijam</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Can we get more logs?</p> <p>I think this could be fixed by patch on bug 21776, just FYI<br/> Liang</p> <p>patch <a href="https://bugzilla.lustre.org/attachment.cgi?id=29521&amp;action=edit" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/attachment.cgi?id=29521&amp;action=edit</a> for 1.8.1 can also be applied on v1_8_3</p> <p>We'd still like to check logs to see whether this issue fits the phenomenon of bz 21776.</p> <p>The customer is happy that it's the above issue and is preparing to update. Unless the problem re-occurs they are going to wait for 1.8.6 and go straight to that.</p> <p>We'll reopen if we observe the issue again once the patch is applied.</p> <p>dup of bug 21776</p> <p>If it is a dup of 21776, note that the o2iblnd does not use PF_MEMALLOC with the patches that were landed to 21776. Also, we (Cray) are chasing a similar instance down attachMD() so other paths might need more help.</p> <p>Re the issues with LNetMDAttach(), ptl_send_rpc() calls libcfs_memory_pressure_get_and_set(), but not until after ptlrpc_register_bulk() which will call LNetMDAttach(). We have seen LNetMDAttach() fail to allocate in that case, which then causes the writeback to hang and deadlock ensues. I'm afraid of creeping the scope where PF_MEMALLOC is used to bail out ptlrpc, however. Comments?</p> <p>o2iblnd will always try to preallocate memory descriptors, so we don't have patch for it. <br/> After a quick look into the patch on BZ, I think the patch to 1.8 is not correct, in ptl_send_rpc:</p> <ul class="alternate" type="square"> <li>2.0: these two lines are before calling of ptlrpc_register_bulk</li> <li>1.8: they are after ptlrpc_register_bulk() which is wrong:</li> </ul> <p> if (request-&gt;rq_memalloc)<br/> mpflag = libcfs_memory_pressure_get_and_set();</p> <p>So I think we can fix it just by moving these two lines ahead.</p> <p>patch tracking at <a href="http://review.whamcloud.com/439" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/439</a></p> <p>1.8.6 needs minor fix</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/17/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,ubuntu1004,inkernel #17</a><br/> <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a> Should let PF_MEMALLOC cover ptlrpc_register_bulk()</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=134461a981b134de896d9aa9cc6ec2d816cfa044" class="external-link" target="_blank" rel="nofollow noopener">134461a981b134de896d9aa9cc6ec2d816cfa044</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/niobuf.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/17/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,el5,inkernel #17</a><br/> <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a> Should let PF_MEMALLOC cover ptlrpc_register_bulk()</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=134461a981b134de896d9aa9cc6ec2d816cfa044" class="external-link" target="_blank" rel="nofollow noopener">134461a981b134de896d9aa9cc6ec2d816cfa044</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/niobuf.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/17/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,client,el6,inkernel #17</a><br/> <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a> Should let PF_MEMALLOC cover ptlrpc_register_bulk()</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=134461a981b134de896d9aa9cc6ec2d816cfa044" class="external-link" target="_blank" rel="nofollow noopener">134461a981b134de896d9aa9cc6ec2d816cfa044</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/niobuf.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/17/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,el6,inkernel #17</a><br/> <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a> Should let PF_MEMALLOC cover ptlrpc_register_bulk()</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=134461a981b134de896d9aa9cc6ec2d816cfa044" class="external-link" target="_blank" rel="nofollow noopener">134461a981b134de896d9aa9cc6ec2d816cfa044</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/niobuf.c</li> </ul> <p>minor fix landed on b1_8</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/17/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,server,el5,inkernel #17</a><br/> <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a> Should let PF_MEMALLOC cover ptlrpc_register_bulk()</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=134461a981b134de896d9aa9cc6ec2d816cfa044" class="external-link" target="_blank" rel="nofollow noopener">134461a981b134de896d9aa9cc6ec2d816cfa044</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/niobuf.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/17/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,el5,ofa #17</a><br/> <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a> Should let PF_MEMALLOC cover ptlrpc_register_bulk()</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=134461a981b134de896d9aa9cc6ec2d816cfa044" class="external-link" target="_blank" rel="nofollow noopener">134461a981b134de896d9aa9cc6ec2d816cfa044</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/niobuf.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/17/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,client,el5,ofa #17</a><br/> <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a> Should let PF_MEMALLOC cover ptlrpc_register_bulk()</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=134461a981b134de896d9aa9cc6ec2d816cfa044" class="external-link" target="_blank" rel="nofollow noopener">134461a981b134de896d9aa9cc6ec2d816cfa044</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/niobuf.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/17/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,server,el5,ofa #17</a><br/> <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a> Should let PF_MEMALLOC cover ptlrpc_register_bulk()</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=134461a981b134de896d9aa9cc6ec2d816cfa044" class="external-link" target="_blank" rel="nofollow noopener">134461a981b134de896d9aa9cc6ec2d816cfa044</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/niobuf.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/17/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,server,el5,ofa #17</a><br/> <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a> Should let PF_MEMALLOC cover ptlrpc_register_bulk()</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=134461a981b134de896d9aa9cc6ec2d816cfa044" class="external-link" target="_blank" rel="nofollow noopener">134461a981b134de896d9aa9cc6ec2d816cfa044</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/niobuf.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/17/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,client,el5,inkernel #17</a><br/> <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a> Should let PF_MEMALLOC cover ptlrpc_register_bulk()</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=134461a981b134de896d9aa9cc6ec2d816cfa044" class="external-link" target="_blank" rel="nofollow noopener">134461a981b134de896d9aa9cc6ec2d816cfa044</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/niobuf.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/17/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,server,el5,inkernel #17</a><br/> <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a> Should let PF_MEMALLOC cover ptlrpc_register_bulk()</p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=134461a981b134de896d9aa9cc6ec2d816cfa044" class="external-link" target="_blank" rel="nofollow noopener">134461a981b134de896d9aa9cc6ec2d816cfa044</a><br/> Files : </p> <ul> <li>lustre/ptlrpc/niobuf.c</li> </ul> <p>Does this fix need landing to master?</p> <p>no, it was already in master.</p> <p>Hi. I agree with the fix to relocate the register bulk. Are you planning on submitting it upstream (to Oracle)?</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/71/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,ubuntu1004,inkernel #71</a><br/> Remove changelog entry for <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a></p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=08b76cd92b2a4b6854ce3910a07531996449a9fd" class="external-link" target="_blank" rel="nofollow noopener">08b76cd92b2a4b6854ce3910a07531996449a9fd</a><br/> Files : </p> <ul> <li>lustre/ChangeLog</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/71/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,el5,ofa #71</a><br/> Remove changelog entry for <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a></p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=08b76cd92b2a4b6854ce3910a07531996449a9fd" class="external-link" target="_blank" rel="nofollow noopener">08b76cd92b2a4b6854ce3910a07531996449a9fd</a><br/> Files : </p> <ul> <li>lustre/ChangeLog</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/71/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,el6,inkernel #71</a><br/> Remove changelog entry for <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a></p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=08b76cd92b2a4b6854ce3910a07531996449a9fd" class="external-link" target="_blank" rel="nofollow noopener">08b76cd92b2a4b6854ce3910a07531996449a9fd</a><br/> Files : </p> <ul> <li>lustre/ChangeLog</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/71/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,client,el5,inkernel #71</a><br/> Remove changelog entry for <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a></p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=08b76cd92b2a4b6854ce3910a07531996449a9fd" class="external-link" target="_blank" rel="nofollow noopener">08b76cd92b2a4b6854ce3910a07531996449a9fd</a><br/> Files : </p> <ul> <li>lustre/ChangeLog</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/71/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,server,el5,ofa #71</a><br/> Remove changelog entry for <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a></p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=08b76cd92b2a4b6854ce3910a07531996449a9fd" class="external-link" target="_blank" rel="nofollow noopener">08b76cd92b2a4b6854ce3910a07531996449a9fd</a><br/> Files : </p> <ul> <li>lustre/ChangeLog</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/71/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,client,el6,inkernel #71</a><br/> Remove changelog entry for <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a></p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=08b76cd92b2a4b6854ce3910a07531996449a9fd" class="external-link" target="_blank" rel="nofollow noopener">08b76cd92b2a4b6854ce3910a07531996449a9fd</a><br/> Files : </p> <ul> <li>lustre/ChangeLog</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/71/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,client,el5,ofa #71</a><br/> Remove changelog entry for <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a></p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=08b76cd92b2a4b6854ce3910a07531996449a9fd" class="external-link" target="_blank" rel="nofollow noopener">08b76cd92b2a4b6854ce3910a07531996449a9fd</a><br/> Files : </p> <ul> <li>lustre/ChangeLog</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/71/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,client,el5,inkernel #71</a><br/> Remove changelog entry for <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a></p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=08b76cd92b2a4b6854ce3910a07531996449a9fd" class="external-link" target="_blank" rel="nofollow noopener">08b76cd92b2a4b6854ce3910a07531996449a9fd</a><br/> Files : </p> <ul> <li>lustre/ChangeLog</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/71/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,server,el5,inkernel #71</a><br/> Remove changelog entry for <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a></p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=08b76cd92b2a4b6854ce3910a07531996449a9fd" class="external-link" target="_blank" rel="nofollow noopener">08b76cd92b2a4b6854ce3910a07531996449a9fd</a><br/> Files : </p> <ul> <li>lustre/ChangeLog</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/71/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » x86_64,server,el5,inkernel #71</a><br/> Remove changelog entry for <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a></p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=08b76cd92b2a4b6854ce3910a07531996449a9fd" class="external-link" target="_blank" rel="nofollow noopener">08b76cd92b2a4b6854ce3910a07531996449a9fd</a><br/> Files : </p> <ul> <li>lustre/ChangeLog</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-b1_8/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/71/" class="external-link" target="_blank" rel="nofollow noopener">lustre-b1_8 » i686,server,el5,ofa #71</a><br/> Remove changelog entry for <a href="https://jira.whamcloud.com/browse/LU-179" title="lustre client lockup when under memory pressure" class="issue-link" data-issue-key="LU-179"><del>LU-179</del></a></p> <p>Johann Lombardi : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/b1_8&amp;a=commit&amp;h=08b76cd92b2a4b6854ce3910a07531996449a9fd" class="external-link" target="_blank" rel="nofollow noopener">08b76cd92b2a4b6854ce3910a07531996449a9fd</a><br/> Files : </p> <ul> <li>lustre/ChangeLog</li> </ul> <p>Client log</p> <p>We've just had a re-occurrence of this problem running 1.8.5.56 (as tagged in git).<br/> Client starts logging problems at Jun 9 14:49:13.</p> <p>Was able to get output from top from the last client lockup; pdflush is sat in 100% CPU.</p> <p>top - 09:10:36 up 2 days, 20:08, 2 users, load average: 801.64, 799.78, 796.51<br/> Tasks: 2891 total, 36 running, 2855 sleeping, 0 stopped, 0 zombie<br/> Cpu(s): 0.0%us, 25.1%sy, 0.0%ni, 70.8%id, 4.1%wa, 0.0%hi, 0.0%si, 0.0%st<br/> Mem: 528386840k total, 70774068k used, 457612772k free, 112k buffers<br/> Swap: 4192924k total, 0k used, 4192924k free, 81176k cached</p> <p> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br/> 13691 cgppipe 39 19 23.6g 23g 9992 S 201 4.6 7646:35 java <br/> 5640 cgppipe 39 19 3122m 2.7g 744 S 100 0.5 5717:22 bwa <br/> 18662 root 0 -20 4 4 0 R 100 0.0 3756:26 elim.uptime <br/> 153 root 20 0 0 0 0 R 100 0.0 3759:05 pdflush <br/> 5528 root 20 0 13992 1528 900 R 100 0.0 3761:22 pim <br/> 1809 root 20 0 56440 7628 2240 R 3 0.0 0:04.24 top <br/> 4612 root 20 0 8832 532 404 S 0 0.0 2:30.10 irqbalance </p> <p>As above the customer is was still observing this problem using the latest code on the 10th Jun, could you reopen this bug accordingly.</p> <p>is it the same pattern as lnet ping hangs while ping works ok?</p> <p>Yes.</p> <p>The system is a NUMA system with 512Gb ram currently. The problem seems to happen during memory pressure, a figure of 70% has been quoted but it's worth saying the application is single-threaded so it's quite likely that some NUMA regions are experiencing 100% memory usage.</p> <p>One thing I've suggested is pinning the application to a different NUMA region to the lustre kernel threads (if this is even possible) so the application wouldn't starve Lustre of memory so easily.</p> <p>can you help checkout what lustre threads was doing during this hang? (better to have thread stacks)</p> <p>We upgraded the kernel on this machine from 2.6.27/SLES11 kernel + 1.8.5.56 lustre client to 2.6.32 kernel +1.8.5.56 lustre client , and the problems seems to have stopped. </p> <p>You can close this issue.</p> <p>Thanks,</p> <p>Guy</p> <p>close ticket per Guy Coates' update info.</p> Development Rank 1|hzw05b: Rank (Obsolete) 10103 Severity