https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5415 Lustre <p>When LRU resizing is enabled on client, sometimes, ldlm_poold have extremely high CPU load. And at the meantime, schedule_timeout() complains about negative timeout. After some time, the problem will recover without any manual intervention. But it happens really frequently when the file system is under high load.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>top - 09:48:51 up 6 days, 11:17, 2 users, load average: 1.00, 1.01, 1.00 Tasks: 516 total, 2 running, 514 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 6.4%sy, 0.0%ni, 93.4%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 65903880k total, 24300068k used, 41603812k free, 346516k buffers Swap: 65535992k total, 0k used, 65535992k free, 18665656k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 37976 root 20 0 0 0 0 R 99.4 0.0 2412:25 ldlm_bl_04 Jul 13 12:49:30 mu01 kernel: LustreError: 11-0: lustre-OST000a-osc-ffff88080fdad800: Communicating with 10.0.2.2@o2ib, operation obd_ping failed with -107. Jul 13 12:49:30 mu01 kernel: Lustre: lustre-OST000a-osc-ffff88080fdad800: Connection to lustre-OST000a (at 10.0.2.2@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 13 12:49:30 mu01 kernel: LustreError: 167-0: lustre-OST000a-osc-ffff88080fdad800: This client was evicted by lustre-OST000a; in progress operations using this service will fail. Jul 13 12:49:31 mu01 kernel: schedule_timeout: wrong timeout value fffffffff5c2c8c0 Jul 13 12:49:31 mu01 kernel: Pid: 4054, comm: ldlm_poold Tainted: G --------------- T 2.6.32-279.el6.x86_64 #1 Jul 13 12:49:31 mu01 kernel: Call Trace: Jul 13 12:49:31 mu01 kernel: [&lt;ffffffff814fe759&gt;] ? schedule_timeout+0x2c9/0x2e0 Jul 13 12:49:31 mu01 kernel: [&lt;ffffffffa086612b&gt;] ? ldlm_pool_recalc+0x10b/0x130 [ptlrpc] Jul 13 12:49:31 mu01 kernel: [&lt;ffffffffa084cfb9&gt;] ? ldlm_namespace_put+0x29/0x60 [ptlrpc] Jul 13 12:49:31 mu01 kernel: [&lt;ffffffffa08670b0&gt;] ? ldlm_pools_thread_main+0x1d0/0x2f0 [ptlrpc] Jul 13 12:49:31 mu01 kernel: [&lt;ffffffff81060250&gt;] ? default_wake_function+0x0/0x20 Jul 13 12:49:31 mu01 kernel: [&lt;ffffffffa0866ee0&gt;] ? ldlm_pools_thread_main+0x0/0x2f0 [ptlrpc] Jul 13 12:49:31 mu01 kernel: [&lt;ffffffff81091d66&gt;] ? kthread+0x96/0xa0 Jul 13 12:49:31 mu01 kernel: [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 Jul 13 12:49:31 mu01 kernel: [&lt;ffffffff81091cd0&gt;] ? kthread+0x0/0xa0 Jul 13 12:49:31 mu01 kernel: [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 Jul 13 12:49:33 mu01 kernel: Lustre: lustre-OST000a-osc-ffff88080fdad800: Connection restored to lustre-OST000a (at 10.0.2.2@o2ib) </pre> </div></div> LU-5415

High ldlm_poold load on client

Bug Critical Resolved Fixed Zhenyu Xu Li Xi patch Fri, 25 Jul 2014 03:14:30 +0000 Thu, 29 Oct 2015 17:25:04 +0000 Thu, 14 Aug 2014 14:05:21 +0000 Lustre 2.6.0 Lustre 2.7.0 Lustre 2.5.3 0 6 <p>We've seen this problem for a long time on variable systems. Ususally, as a walk around, we disable LRU resizing of ldlm on client. But maybe following patch can help.</p> <p><a href="http://review.whamcloud.com/11227" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11227</a></p> <p>Lai</p> <p>Could you please review this patch?</p> <p>Thanks</p> <p>Peter</p> <p>Was this problem actually seen on Lustre 2.6/master or some other version? There were patches from Oleg that were landed for 2.5 that addressed some problems with LDLM pools, but I'm happy to see more improvements in this area. </p> <p><a href="http://review.whamcloud.com/6234" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6234</a><br/> <a href="http://review.whamcloud.com/5793" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5793</a><br/> <a href="http://review.whamcloud.com/5624" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5624</a></p> <p>So the patches Andreas mentions would be included on any 2.5.x based branches.</p> <p>Yeah, those patches are included on the branch which has this problem. This problem is happening quite frequently,</p> <p>Peter, I'll be on vacation from tomorrow, could you reassign to others?</p> <p>Bobijam</p> <p>Could you please look after this patch?</p> <p>Thanks</p> <p>Peter</p> <p>I wonder what sort of lists do you have on the client side that cause iteration of said lists to take over a second (so that the time becomes negative)? Could the problem be somewhere else and this proposed change is just papering over the real issue?</p> <p>Yeah, that is very possible that the patch is not fixing the root cause. And it nearly becomes a common knowledge that LRU resizing of ldlm on client should be disabled, otherwise there will be ldlm high load. Is there any garantee that LRU resizing will complete in a determined period of time? If not, then the current codes has problem any way. And I found some other issues which had negative timeout values.</p> <p><a href="https://jira.hpdd.intel.com/browse/LU-1733" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-1733</a><br/> <a href="https://jira.hpdd.intel.com/browse/LU-4480" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-4480</a><br/> <a href="https://jira.hpdd.intel.com/browse/DDN-58" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/DDN-58</a></p> <p>Patch for b2_5:<br/> <a href="http://review.whamcloud.com/11414" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11414</a></p> <p>Landed for 2.7</p> Related LU-2924 LU-4536 Development Rank 1|hzwsan: Rank (Obsolete) 15059 Severity