https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13413 Lustre <p>Currently our production file system is experience peer credit exhaustion which is leading to soft locks. The back trace are attached. Instead of a soft lockups we should be having evicts.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[2642432.333239] [&lt;ffffffff9477544a&gt;] queued_spin_lock_slowpath+0xb/0xf [2642432.333241] [&lt;ffffffff94783330&gt;] _raw_spin_lock+0x20/0x30 [2642432.333258] [&lt;ffffffffc146202c&gt;] lock_res_and_lock+0x2c/0x50 [ptlrpc] [2642432.333273] [&lt;ffffffffc1469c61&gt;] ldlm_lock_enqueue+0x1b1/0xa20 [ptlrpc] [2642432.333294] [&lt;ffffffffc14b9891&gt;] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [2642432.333311] [&lt;ffffffffc1492506&gt;] ldlm_handle_enqueue0+0xa56/0x15f0 [ptlrpc] [2642432.333332] [&lt;ffffffffc14bb300&gt;] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [2642432.333357] [&lt;ffffffffc151acf2&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] [2642432.333381] [&lt;ffffffffc1521b0a&gt;] tgt_request_handle+0xada/0x1570 [ptlrpc] [2642432.333404] [&lt;ffffffffc14fb021&gt;] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [2642432.333408] [&lt;ffffffffc103fbde&gt;] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [2642432.333428] [&lt;ffffffffc14c646b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [2642432.333449] [&lt;ffffffffc14c3285&gt;] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [2642432.333451] [&lt;ffffffff940d3903&gt;] ? __wake_up+0x13/0x20 [2642432.333471] [&lt;ffffffffc14c9dd4&gt;] ptlrpc_main+0xb34/0x1470 [ptlrpc] </pre> </div></div> <p> </p> RHEL7.7 running Lustre 2.12 LTS LU-13413

Lustre soft lockups with peer credit exhaustion

Bug Major Resolved Fixed Andreas Dilger James A Simmons ORNL Fri, 3 Apr 2020 13:53:07 +0000 Mon, 11 Jul 2022 23:56:41 +0000 Thu, 2 Dec 2021 16:53:45 +0000 Lustre 2.12.4 0 9 <p>Hi James, any changes to the system that would have triggered this to start happening (upgraded Lustre, new patch, new clients, new workload, etc.)?</p> <p>I talked to our admin team and they stated its the same problem since October with the upgrade to 2.12 LTS. With <a href="https://jira.whamcloud.com/browse/LU-12600" title="Lustre tgt_brw_write() bug" class="issue-link" data-issue-key="LU-12600"><del>LU-12600</del></a> it did reduce the problem but it still exist. This problem shows up when clients are hitting the servers with large numbers of small I/Os.</p> <p>Do you think tuning the lru_resize and lru_size might help? The current values are</p> <p>lru_size=2000 and lru_resize=3900000</p> <p>James, how many locks are actually being held by the clients and are on the servers? Running "<tt>lctl get_param ldlm.namespaces.&#42;.lock_count</tt>" on some clients and servers would give a good idea of where this is at. There were a couple of bugs related to not canceling the DLM locks on the clients, even above LRU size, so it would be good to know if this is the problem you are seeing.</p> <p>Also, are there any patches above 2.12.4?</p> <p>The only patch we are running outside of 2.12 is a port of <a href="https://jira.whamcloud.com/browse/LU-13101" title="Eviction during ll_open_cleanup()" class="issue-link" data-issue-key="LU-13101"><del>LU-13101</del></a>.</p> <p>f2-oss1c1: ldlm.namespaces.filter-f2-OST0002_UUID.lock_count=13071<br/> f2-oss1c1: ldlm.namespaces.filter-f2-OST0026_UUID.lock_count=15371<br/> f2-oss1f5: ldlm.namespaces.filter-f2-OST001d_UUID.lock_count=9772<br/> f2-oss1f5: ldlm.namespaces.filter-f2-OST0041_UUID.lock_count=23522<br/> f2-oss1f4: ldlm.namespaces.filter-f2-OST0017_UUID.lock_count=6410<br/> f2-oss1f4: ldlm.namespaces.filter-f2-OST003b_UUID.lock_count=10967<br/> f2-oss1e2: ldlm.namespaces.filter-f2-OST000a_UUID.lock_count=9424<br/> f2-oss1e2: ldlm.namespaces.filter-f2-OST002e_UUID.lock_count=14095<br/> f2-oss1e6: ldlm.namespaces.filter-f2-OST0022_UUID.lock_count=9126<br/> f2-oss1e6: ldlm.namespaces.filter-f2-OST0046_UUID.lock_count=14720<br/> f2-oss1e1: ldlm.namespaces.filter-f2-OST0004_UUID.lock_count=5522<br/> f2-oss1e1: ldlm.namespaces.filter-f2-OST0028_UUID.lock_count=13113<br/> f2-oss1c4: ldlm.namespaces.filter-f2-OST0014_UUID.lock_count=31000<br/> f2-oss1c4: ldlm.namespaces.filter-f2-OST0038_UUID.lock_count=8096<br/> f2-oss1a1: ldlm.namespaces.filter-f2-OST0000_UUID.lock_count=9730<br/> f2-oss1a1: ldlm.namespaces.filter-f2-OST0024_UUID.lock_count=10386<br/> f2-oss1a2: ldlm.namespaces.filter-f2-OST0006_UUID.lock_count=6744<br/> f2-oss1a2: ldlm.namespaces.filter-f2-OST002a_UUID.lock_count=17169<br/> f2-oss1b2: ldlm.namespaces.filter-f2-OST0007_UUID.lock_count=7458<br/> f2-oss1b2: ldlm.namespaces.filter-f2-OST002b_UUID.lock_count=12837<br/> f2-oss1e3: ldlm.namespaces.filter-f2-OST0010_UUID.lock_count=17431<br/> f2-oss1e3: ldlm.namespaces.filter-f2-OST0034_UUID.lock_count=11464<br/> f2-oss1d4: ldlm.namespaces.filter-f2-OST0015_UUID.lock_count=9836<br/> f2-oss1d4: ldlm.namespaces.filter-f2-OST0039_UUID.lock_count=14168<br/> f2-oss1d3: ldlm.namespaces.filter-f2-OST000f_UUID.lock_count=11082<br/> f2-oss1d3: ldlm.namespaces.filter-f2-OST0033_UUID.lock_count=9801<br/> f2-oss1f6: ldlm.namespaces.filter-f2-OST0023_UUID.lock_count=7558<br/> f2-oss1f6: ldlm.namespaces.filter-f2-OST0047_UUID.lock_count=10884<br/> f2-oss1c6: ldlm.namespaces.filter-f2-OST0020_UUID.lock_count=11754<br/> f2-oss1c6: ldlm.namespaces.filter-f2-OST0044_UUID.lock_count=21609<br/> f2-oss1f2: ldlm.namespaces.filter-f2-OST000b_UUID.lock_count=7273<br/> f2-oss1f2: ldlm.namespaces.filter-f2-OST002f_UUID.lock_count=16546<br/> f2-oss1d2: ldlm.namespaces.filter-f2-OST0009_UUID.lock_count=5351<br/> f2-oss1d2: ldlm.namespaces.filter-f2-OST002d_UUID.lock_count=11648<br/> f2-oss1b4: ldlm.namespaces.filter-f2-OST0013_UUID.lock_count=7091<br/> f2-oss1b4: ldlm.namespaces.filter-f2-OST0037_UUID.lock_count=17672<br/> f2-oss1d1: ldlm.namespaces.filter-f2-OST0003_UUID.lock_count=10786<br/> f2-oss1d1: ldlm.namespaces.filter-f2-OST0027_UUID.lock_count=12821<br/> f2-oss1a3: ldlm.namespaces.filter-f2-OST000c_UUID.lock_count=8583<br/> f2-oss1a3: ldlm.namespaces.filter-f2-OST0030_UUID.lock_count=10223<br/> f2-oss1b1: ldlm.namespaces.filter-f2-OST0001_UUID.lock_count=19223<br/> f2-oss1b1: ldlm.namespaces.filter-f2-OST0025_UUID.lock_count=12262<br/> f2-oss1c3: ldlm.namespaces.filter-f2-OST000e_UUID.lock_count=19796<br/> f2-oss1c3: ldlm.namespaces.filter-f2-OST0032_UUID.lock_count=19187<br/> f2-oss1c2: ldlm.namespaces.filter-f2-OST0008_UUID.lock_count=22208<br/> f2-oss1c2: ldlm.namespaces.filter-f2-OST002c_UUID.lock_count=17602<br/> f2-oss1e5: ldlm.namespaces.filter-f2-OST001c_UUID.lock_count=10177<br/> f2-oss1e5: ldlm.namespaces.filter-f2-OST0040_UUID.lock_count=18898<br/> f2-oss1d6: ldlm.namespaces.filter-f2-OST0021_UUID.lock_count=9237<br/> f2-oss1d6: ldlm.namespaces.filter-f2-OST0045_UUID.lock_count=23232<br/> f2-oss1b6: ldlm.namespaces.filter-f2-OST001f_UUID.lock_count=8040<br/> f2-oss1b6: ldlm.namespaces.filter-f2-OST0043_UUID.lock_count=22877<br/> f2-oss1f1: ldlm.namespaces.filter-f2-OST0005_UUID.lock_count=8117<br/> f2-oss1f1: ldlm.namespaces.filter-f2-OST0029_UUID.lock_count=9207<br/> f2-oss1b5: ldlm.namespaces.filter-f2-OST0019_UUID.lock_count=12895<br/> f2-oss1b5: ldlm.namespaces.filter-f2-OST003d_UUID.lock_count=25789</p> <blockquote> <p>f2-oss1f1: ldlm.namespaces.filter-f2-OST0029_UUID.lock_count=9207<br/> f2-oss1b5: ldlm.namespaces.filter-f2-OST0019_UUID.lock_count=12895<br/> f2-oss1b5: ldlm.namespaces.filter-f2-OST003d_UUID.lock_count=25789</p></blockquote> <p>OK, so this is totally reasonable, even low by my expectation, so it doesn't seem like the spinlock hold times are driven by having too many DLM locks to process, unless there is a sudden spike of locks at the time the problem occurs?</p> <p>According to the stack traces, there are only really a few "interesting" process, the rest are just blocked on (likely) the same spinlock in either <tt>ldlm_lock_enqueue()</tt> or <tt>ofd_lvbo_fill()</tt>. In both cases, it looks like the <tt>ldlm_cn</tt> threads are processing a potentially long list of locks on a single resource:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[2642432.309284] CPU: 8 PID: 57948 Comm: ldlm_cn00_044 3.10.0-1062.9.1.el7.x86_64 [2642432.309314] Call Trace: [2642432.309331] [&lt;ffffffffc14808e3&gt;] ldlm_process_extent_lock+0x213/0x490 [ptlrpc] [2642432.309362] [&lt;ffffffffc146959e&gt;] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [2642432.309383] [&lt;ffffffffc146a5d2&gt;] __ldlm_reprocess_all+0x102/0x360 [ptlrpc] [2642432.309397] [&lt;ffffffffc146a843&gt;] ldlm_reprocess_all+0x13/0x20 [ptlrpc] [2642432.309414] [&lt;ffffffffc148f4df&gt;] ldlm_request_cancel+0x42f/0x780 [ptlrpc] [2642432.309431] [&lt;ffffffffc14937a2&gt;] ldlm_handle_cancel+0x232/0x2b0 [ptlrpc] [2642432.309447] [&lt;ffffffffc1493978&gt;] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [2642432.309467] [&lt;ffffffffc14c646b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [2642432.309509] [&lt;ffffffffc14c9dd4&gt;] ptlrpc_main+0xb34/0x1470 [ptlrpc] [2642432.309530] [&lt;ffffffff940c61f1&gt;] kthread+0xd1/0xe0 [2644718.775432] CPU: 15 PID: 52157 Comm: ldlm_cn01_021 3.10.0-1062.9.1.el7.x86_64 [2644718.775459] Call Trace: [2644718.775477] [&lt;ffffffffc14808e3&gt;] ldlm_process_extent_lock+0x213/0x490 [ptlrpc] [2644718.775508] [&lt;ffffffffc146959e&gt;] ldlm_reprocess_queue+0x1be/0x3f0 [ptlrpc] [2644718.775528] [&lt;ffffffffc146a5d2&gt;] __ldlm_reprocess_all+0x102/0x360 [ptlrpc] [2644718.775543] [&lt;ffffffffc146a843&gt;] ldlm_reprocess_all+0x13/0x20 [ptlrpc] [2644718.775559] [&lt;ffffffffc148f4df&gt;] ldlm_request_cancel+0x42f/0x780 [ptlrpc] [2644718.775576] [&lt;ffffffffc14937a2&gt;] ldlm_handle_cancel+0x232/0x2b0 [ptlrpc] [2644718.775593] [&lt;ffffffffc1493978&gt;] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [2644718.775613] [&lt;ffffffffc14c646b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [2644718.775656] [&lt;ffffffffc14c9dd4&gt;] ptlrpc_main+0xb34/0x1470 [ptlrpc] [2644718.775677] [&lt;ffffffff940c61f1&gt;] kthread+0xd1/0xe0 </pre> </div></div> <p>I would guess there is some application which is "piling on" a single object and getting lots of conflicting locks (e.g. high IOPS small/unaligned writes on a HDD OST) that generates a lot of conflicting locks, then repeatedly reprocessing the lock each time a new one is granted/released? I was wondering whether patch <a href="https://review.whamcloud.com/33221" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33221</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-11085" title="Replace Lustre interval tree with kernel one" class="issue-link" data-issue-key="LU-11085">LU-11085</a> ldlm: simplify use of interval-tree</tt>" had already landed and might be causing this, but I see it has not. It <em>seems</em> like the interval tree should protect us against scanning a lot of non-overlapping locks, so possibly there are a lot of overlapping lock requests from the application?</p> <p>Andreas,</p> <p>           I wanted to follow up with you on this since it has been a while. Generally the workload that triggers this happens during the first week of the month (a specific code runs by a subset of users on a set schedule). Based on your input about conflicting locks, we changed the lru_size configuration on the compute nodes to limit to 200 instead of the dynamic setting. We were thinking that this would limit the number of metadata transactions a single compute node could issue concurrently to the MDS. After this change we did not see more crashes, but it is possible this will crop up here again in the next week.</p> <p> </p> <p>Thanks,</p> <p>Dustin</p> <p>Dustin, James, it looks like this problem has not been hit again since the <tt>lru_size=200</tt> change was made back in April. Were you ever able to track down if there was a specific workload that was causing the high lock traffic? </p> Related LU-12600 LU-14221 Development Rank 1|i00wxb: Rank (Obsolete) 9223372036854775807 Severity