https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16939 Lustre <p>We already have the lock contention detection patch: <a href="https://review.whamcloud.com/35287/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35287/</a></p> <p>With lock contention detection, the lock server will mark the lock resource (a shared OST object) as contention in a time period (2 seconds, by default). And the client requested a extent lock will be informed the lock contention.</p> <p>Under lock contention, the lock server will not expand the lock extent to avoid unnecessary lock conflict callbacks.<br/> And the client could be switch from buffered I/O to the direct I/O if the I/O size is large enough.</p> LU-16939

ldlm: not expand the lock extent when the shared file is under lock contention

Improvement Minor Open Unresolved Qian Yingjin Qian Yingjin Tue, 4 Jul 2023 09:42:14 +0000 Mon, 22 Jan 2024 16:29:55 +0000 1 3 <p>I'm glad you found <a href="https://jira.whamcloud.com/browse/LU-12550" title="automatic lockahead" class="issue-link" data-issue-key="LU-12550">LU-12550</a>, that's important.</p> <p>The switch from buffered to direct is good too - I was planning to do that eventually using <a href="https://jira.whamcloud.com/browse/LU-12550" title="automatic lockahead" class="issue-link" data-issue-key="LU-12550">LU-12550</a>.  We do need unaligned DIO first.  I have some more thoughts on this specific ticket which I'll put here in a moment.</p> <p>So, I think we have to be careful here.  When I first developed lockahead, I tested turning off lock expansion when doing multi-client shared file.  So exactly what you're suggesting here.</p> <p>I found it <b>hurt</b> performance in the shared file case.</p> <p>The reason is a little challenging to explain (and the graphs I made for this are lost long ago, heh).</p> <p>Normally, when (for example) two clients are writing a single OST object with one process each, we imagine the process is something like this.</p> <p>Client 1 writes [0, 1 MiB).  Requests lock, call it lock 'A'.  Server expands lock A, 0 - infinity.<br/> Client 2 writes [1,2 MiB).  Requests lock B.  Server cancels lock A.  Server expands lock B, 0 - infinity.  (Or maybe it's 1 MiB to infinity?  Doesn't matter)<br/> Client 1 writes [2, 3 Mib).  Requests lock C...  etc, etc.</p> <p>Here is what actually happens.<br/> Client 1 writes [0, 1 MiB).  Acquires lock A, 0 - infinity.</p> <p>Client 2 tries to write [1, 2 MiB), sends request to server, etc.<br/> Client 1 writes [2, 3 MiB) using lock A.</p> <p>Client 1 writes [4, 5 MiB) using lock A...</p> <p>Client 1 writes [6. 7 MiB) using lock A...<br/> <span class="error">&#91;...&#93;</span></p> <p>Server calls back lock A, but only after client 1 has done some number of writes.  (In my testing, I think it was like 4 1 MiB writes on average?)</p> <p>So client 2 is waiting for client 1, but client 1 gets a bunch of work done before the lock is called back.</p> <p>This is very important, <b>because</b> if you turn off lock expansion, now the client write process looks like this.</p> <p>Start write</p> <p>Request lock</p> <p><b>Lock request goes to server</b></p> <p><b>Get lock from server</b></p> <p>Do write...</p> <p>We have added an extra round trip to the server for every write.</p> <p>So what I found in my testing was that until contention was very high (many clients to a single object), the overhead of doing a lock request for <b>every</b> write was worse than waiting for the other client.  Note if there is &gt; 1 process writing to each OST object on the same client, they also benefit from sharing the expanded lock.  So disabling lock expansion on contention often made performance <b>worse,</b> because the overhead of asking for a lock for every write was worse than the serialization of waiting for the other client(s).</p> <p>This is just something to be aware of and test for.  My testing was on disk, flash may change things.  I think disabling expansion will help under heavy contention, but only under heavy contention (many clients to one object).  So we should do some careful testing of write performance with disabling expansion on contention, with both a small and a large number of clients <b>and</b> processes per object.</p> <p>There is an OST tunable <tt>ldlm.namespaces.*.contended_locks</tt> that controls how many lockers are needed on a resource before it considers the file "contended". IIRC, once there were more than 4 contending clients the lock would not grow downward, and after <tt>contended_locks=32</tt> contending clients the lock would also not grow upward. Even under some contention, if the client is doing e.g. 16MB writes in a 1GB section per process it still makes sense to expand the DLM lock up to the start of the next extent (i.e. 1GB boundary) so that it doesn't have to get a separate lock for each write.</p> <p>I think the right solution is to use the DLM contention information to do lockless (sync+direct) writes instead of getting any DLM locks at all. At that point the many clients writing to the file should be saturating the OSS so it will have enough outstanding RPCs to fill its pipeline and the clients are blocked on sending so the sync writes will not really hurt performance. That also allows the OST to merge the writes during submission instead of doing read-modify-write.</p> <p>Patrick, Andreas,<br/> Thanks for your helpful comments!</p> <p>I am just worried about the lock contention accounting in the current implementation maybe not correct.<br/> We clear the contention locks only when the contention information on the resource is older than the time window (2s):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* If our contention info is older than the time window, clear it */</span> <span class="code-keyword">if</span> (now &gt; res-&gt;lr_contention_age + ns-&gt;ns_contention_time) res-&gt;lr_contended_locks = 0; </pre> </div></div> <p>For example, <br/> time wind: lock contention count: account:<br/> In [0s, 2s) - lock contents count: 32 @lr_contended_locks = 32<br/> In [2s, 4s) - lock contents count: 1 @lr_contended_locks = 33<br/> In [4s, 6s) - lock contents count: 1 @lr_contended_locks = 34<br/> In [6s, 8s) - lock contents count: 1 @lr_contended_locks = 35<br/> ...</p> <p>The accounting contended locks is not decaying even with low lock contention conflict.</p> <p>So I am thinking about improving the accounting for lock contention based on time windows (similar to heat):<br/> obd_heat_decay();<br/> obd_heat_add();<br/> obd_heat_get();</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> * * The file heat is calculated <span class="code-keyword">for</span> every time interval period I. The access * frequency during each period is counted. The file heat is only recalculated * at the end of a time period. And a percentage of the former file heat is * lost when recalculated. The recursion formula to calculate the heat of the * file f is as follow: * * Hi+1(f) = (1-P)*Hi(f)+ P*Ci * * Where Hi is the heat value in the period between time points i*I and * (i+1)*I; Ci is the access count in the period; the symbol P refers to the * weight of Ci. The larger the value the value of P is, the more influence Ci * has on the file heat. */ </pre> </div></div> <p>What's your opioion?</p> <p>Thanks,<br/> Qian</p> <p>I found two URLs that can be used for the lock contention rate calculation based on fixed time sliding window:<br/> <a href="https://medium.com/@avocadi/rate-limiter-sliding-window-counter-7ec08dbe21d6" class="external-link" target="_blank" rel="nofollow noopener">https://medium.com/@avocadi/rate-limiter-sliding-window-counter-7ec08dbe21d6</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> previous_window_request_count = 8 current_window_request_count = 5 previous_window_count_weight = 0.47 Hence, 8 * 0.47 + 5 = 8.76 </pre> </div></div> <p><a href="https://docs.konghq.com/gateway/latest/kong-plugins/rate-limiting/algorithms/rate-limiting/" class="external-link" target="_blank" rel="nofollow noopener">https://docs.konghq.com/gateway/latest/kong-plugins/rate-limiting/algorithms/rate-limiting/</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> current window rate: 10 previous window rate: 40 window size: 60 window position: 30 (seconds past the start of the current window) weight = .5 (60 second window size - 30 seconds past the window start) rate = <span class="code-quote">'current rate'</span> + <span class="code-quote">'previous weight'</span> * <span class="code-quote">'weight'</span> = 10 + 40 * (<span class="code-quote">'window size'</span> - <span class="code-quote">'window position'</span>) / <span class="code-quote">'window size'</span> = 10 + 40 * (60 - 30) / 60 = 10 + 40 * .5 = 30 the formula used to define the weighting percentage is as follows: weight = (window_size - (time() % window_size)) / window_size </pre> </div></div> <p>I think a simple decaying average is probably OK, and the contention should go to zero if there are no new locks within 2-3 decay intervals.</p> <p>Hi Andreas,</p> <p>Could you please give out the detailed formula or algorithm for the decaying calculation?</p> <p> </p> <p>The above gives our two decaying algorithm based on the fixed time-based sliding windows.</p> <p>The time window length is 2:</p> <ul> <li>previous_count</li> <li>current_count</li> </ul> <p>A. Fixed decaying weight for the previous_count: i.e. previous_count_decay_weight = 0.5<br/> rate = current_count + previous_count * previous_count_decay_weight;</p> <p>B. dynamic decaying weight for the previous_count based on the current time pos in the current time winodw:<br/> rate = current_count + previous_count * (window_size - (time() % window_size)) / window_size;</p> <p>Thanks,<br/> Qian</p> <p>        </p> <blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>previous_count_decay_weight = 0.5 rate = current_count + previous_count * previous_count_decay_weight; </pre> </div></div></blockquote> <p>In this formula, if current_count = previous_count, then "<tt>rate = 1.5 x current_count</tt>" which is bad. </p> <p>I think one of the important properties of decaying average is that they are equal to the original value if it is constant. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>decay_weight = 0.3 new_value = current_value * (1 - decay_weight) + previous_value * decay_weight </pre> </div></div> <p>That means that the weighted sum = 1.0 no matter what the weight is, and if <tt>previous_value = current_value</tt>, then <tt>new_value</tt> will be the same. </p> <p>In the above formulas </p> Related LU-12550 LU-16669 Development Rank 1|i03pj3: Rank (Obsolete) 9223372036854775807 Severity