https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6390 Lustre <p>Here is all resutls results with 200K files.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl set_param ldlm.namespaces.*.lru_size=1000 ldlm.namespaces.MGC10.0.10.153@o2ib.lru_size=1000 ldlm.namespaces.lustre-MDT0000-mdc-ffff881fccd75800.lru_size=1000 ldlm.namespaces.lustre-OST0000-osc-ffff881fccd75800.lru_size=1000 ldlm.namespaces.lustre-OST0001-osc-ffff881fccd75800.lru_size=1000 ldlm.namespaces.lustre-OST0002-osc-ffff881fccd75800.lru_size=1000 ldlm.namespaces.lustre-OST0003-osc-ffff881fccd75800.lru_size=1000 # ls -lR /lustre # lctl get_param ldlm.namespaces.*.lock_count ldlm.namespaces.MGC10.0.10.153@o2ib.lock_count=4 ldlm.namespaces.lustre-MDT0000-mdc-ffff881fccd75800.lock_count=1002 ldlm.namespaces.lustre-OST0000-osc-ffff881fccd75800.lock_count=50003 ldlm.namespaces.lustre-OST0001-osc-ffff881fccd75800.lock_count=50002 ldlm.namespaces.lustre-OST0002-osc-ffff881fccd75800.lock_count=50003 ldlm.namespaces.lustre-OST0003-osc-ffff881fccd75800.lock_count=50004 </pre> </div></div> LU-6390

lru_size on the OSC is not honored

Bug Critical Resolved Fixed Jinshan Xiong Jinshan Xiong patch Thu, 19 Mar 2015 23:34:14 +0000 Thu, 1 Nov 2018 07:28:34 +0000 Fri, 19 Jun 2015 12:01:18 +0000 Lustre 2.8.0 Lustre 2.8.0 0 11 <p>Vitaly Fertman (vitaly_fertman@xyratex.com) uploaded a new patch: <a href="http://review.whamcloud.com/14342" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14342</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6390" title="lru_size on the OSC is not honored" class="issue-link" data-issue-key="LU-6390"><del>LU-6390</del></a> ldlm: restore the ELC for enqueue<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3f9b5d1aea04d30a76f874edc5048689cd98308a</p> <p>Customer site installed <a href="http://review.whamcloud.com/14342" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14342</a> on their clients. After ~6 hours of running, a data mover client hung. The node is not configured to take memory dumps so we captured stack traces from /proc. The stack traces are reminiscent of <a href="https://jira.whamcloud.com/browse/LU-4300" title="ptlrpcd threads deadlocked in cl_lock_mutex_get" class="issue-link" data-issue-key="LU-4300"><del>LU-4300</del></a>. 11 out of 32 ptlrcpd threads and 47 out of 60 ldlm_bl threads are waiting in cl_lock_mutex_get. And 2 ll_agl threads are stuck in osc_extent_wait:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>10730 ll_agl_21508 11228 ll_agl_21487 [&lt;ffffffffa0f4eb90&gt;] osc_extent_wait+0x420/0x670 [osc] [&lt;ffffffffa0f4f0af&gt;] osc_cache_wait_range+0x2cf/0x890 [osc] [&lt;ffffffffa0f50281&gt;] osc_cache_writeback_range+0xc11/0xfb0 [osc] [&lt;ffffffffa0f3b6f4&gt;] osc_lock_flush+0x84/0x280 [osc] [&lt;ffffffffa0f3b9d7&gt;] osc_lock_cancel+0xe7/0x1c0 [osc] [&lt;ffffffffa0b4cbf5&gt;] cl_lock_cancel0+0x75/0x160 [obdclass] [&lt;ffffffffa0b4d7ab&gt;] cl_lock_cancel+0x13b/0x140 [obdclass] [&lt;ffffffffa0f3cf1a&gt;] osc_ldlm_blocking_ast+0x13a/0x350 [osc] [&lt;ffffffffa0cf703c&gt;] ldlm_cancel_callback+0x6c/0x1a0 [ptlrpc] [&lt;ffffffffa0d06eaa&gt;] ldlm_cli_cancel_local+0x8a/0x470 [ptlrpc] [&lt;ffffffffa0d0a1ae&gt;] ldlm_cli_cancel_list_local+0xee/0x290 [ptlrpc] [&lt;ffffffffa0d0b055&gt;] ldlm_cancel_lru_local+0x35/0x40 [ptlrpc] [&lt;ffffffffa0d0c4cc&gt;] ldlm_prep_elc_req+0x3ec/0x4b0 [ptlrpc] [&lt;ffffffffa0d0c5b8&gt;] ldlm_prep_enqueue_req+0x28/0x30 [ptlrpc] [&lt;ffffffffa0f205d9&gt;] osc_enqueue_base+0x109/0x5a0 [osc] [&lt;ffffffffa0f3c5cd&gt;] osc_lock_enqueue+0x1ed/0x890 [osc] [&lt;ffffffffa0b50c2c&gt;] cl_enqueue_try+0xfc/0x300 [obdclass] [&lt;ffffffffa0fce64a&gt;] lov_lock_enqueue+0x21a/0xf10 [lov] [&lt;ffffffffa0b50c2c&gt;] cl_enqueue_try+0xfc/0x300 [obdclass] [&lt;ffffffffa0b51b4f&gt;] cl_enqueue_locked+0x6f/0x1f0 [obdclass] [&lt;ffffffffa0b5279e&gt;] cl_lock_request+0x7e/0x270 [obdclass] [&lt;ffffffffa109d000&gt;] cl_glimpse_lock+0x180/0x490 [lustre] [&lt;ffffffffa109d875&gt;] cl_glimpse_size0+0x1a5/0x1d0 [lustre] [&lt;ffffffffa1095ffb&gt;] ll_agl_trigger+0x1db/0x4b0 [lustre] [&lt;ffffffffa1096e6e&gt;] ll_agl_thread+0x15e/0x490 [lustre] [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff 6143 ptlrpcd_0 + 10 more ptlrpcd threads (out of 32) [&lt;ffffffffa0b4e6df&gt;] cl_lock_mutex_get+0x6f/0xd0 [obdclass] [&lt;ffffffffa0fd5b19&gt;] lovsub_parent_lock+0x49/0x120 [lov] [&lt;ffffffffa0fd6c4f&gt;] lovsub_lock_modify+0x7f/0x1e0 [lov] [&lt;ffffffffa0b4e108&gt;] cl_lock_modify+0x98/0x310 [obdclass] [&lt;ffffffffa0f3de32&gt;] osc_lock_granted+0x1e2/0x2b0 [osc] [&lt;ffffffffa0f3e308&gt;] osc_lock_upcall+0x408/0x600 [osc] [&lt;ffffffffa0f1e7a6&gt;] osc_enqueue_fini+0x106/0x240 [osc] [&lt;ffffffffa0f23272&gt;] osc_enqueue_interpret+0xe2/0x1e0 [osc] [&lt;ffffffffa0d2487c&gt;] ptlrpc_check_set+0x2bc/0x1b50 [ptlrpc] [&lt;ffffffffa0d500cb&gt;] ptlrpcd_check+0x53b/0x560 [ptlrpc] [&lt;ffffffffa0d5071b&gt;] ptlrpcd+0x33b/0x3f0 [ptlrpc] [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff 6209 ldlm_bl_00 + 46 other ldlm_bl threads (out of 60) [&lt;ffffffffa0b4e6df&gt;] cl_lock_mutex_get+0x6f/0xd0 [obdclass] [&lt;ffffffffa0f3ce5a&gt;] osc_ldlm_blocking_ast+0x7a/0x350 [osc] [&lt;ffffffffa0d0f0c0&gt;] ldlm_handle_bl_callback+0x130/0x400 [ptlrpc] [&lt;ffffffffa0d0f5f1&gt;] ldlm_bl_thread_main+0x261/0x3c0 [ptlrpc] [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> <p>I'll attach the complete list of stack traces to this ticket.</p> <p>Let me know whether you need a dump and I'll see if we can reproduce the bug on a test system.</p> <p>ps output followed by <br/> for each pid in /proc; do cat /proc/$pid/stack; done</p> <p>Unique stack traces for Lustre processes extracted from bt.all</p> <p>dmesg file from the data mover node. Shows partial output from <br/> echo t &gt; /proc/sysrq-trigger</p> <p>The forced stack trace dump was done at least 30 minutes before the /proc/pid/stack output was captured. So comparisons between dmesg and bt.all show that the threads are indeed hung.</p> <p>cl_lock_mutex_get does not exists in 2.8, since the CLIO simplification, so this was not 2.8 lustre what you tested.</p> <p>the patch by itself is supposed to be correct, I think the problem is related to the issue rased in <a href="https://jira.whamcloud.com/browse/LU-5781" title="endless loop in osc_lock_weight()" class="issue-link" data-issue-key="LU-5781"><del>LU-5781</del></a>, cancel_lru_policy is called not atomically with set_cbpending, so some dirty pages could be added in between.</p> <p>Sorry, forgot to mention, the client is running 2.5.1 on CentOS 2.6.32-431.20.3.el6.x86_64. (Same system as in LELUS-294).</p> <p>So are you saying that the <a href="https://jira.whamcloud.com/browse/LU-5781" title="endless loop in osc_lock_weight()" class="issue-link" data-issue-key="LU-5781"><del>LU-5781</del></a> patch is needed in addition to <a href="https://jira.whamcloud.com/browse/LU-6390" title="lru_size on the OSC is not honored" class="issue-link" data-issue-key="LU-6390"><del>LU-6390</del></a>? And that the 2 together will fix the problems with too many locks in the LRU? Certainly would explain why we didn't see the <a href="https://jira.whamcloud.com/browse/LU-4300" title="ptlrpcd threads deadlocked in cl_lock_mutex_get" class="issue-link" data-issue-key="LU-4300"><del>LU-4300</del></a>-like hang in our internal testing, since the versions we typically run have both patches.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/14342/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14342/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6390" title="lru_size on the OSC is not honored" class="issue-link" data-issue-key="LU-6390"><del>LU-6390</del></a> ldlm: restore the ELC for enqueue<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ac5abd46e95edd97316ff0e9563288636e7c42bc</p> <p>Landed for 2.8</p> Related LU-6529 LU-11518 Development Rank 1|hzx8zj: Rank (Obsolete) 9223372036854775807 Severity