https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2576 Lustre <p>We've had reports of IO writes hanging in Sequoia, and after some initial debugging, have narrowed the hung threads to getting stuck with the following stack trace:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> sysiod S 00000fffa842633c 0 27910 3135 0x00000000 Call Trace: [c0000003ee34e480] [c0000003ee34e590] 0xc0000003ee34e590 (unreliable) [c0000003ee34e650] [c000000000009b8c] .__switch_to+0xc4/0x100 [c0000003ee34e6e0] [c0000000004364c8] .schedule+0x858/0x9c0 [c0000003ee34e990] [8000000000a720c0] .cfs_waitq_wait+0x10/0x30 [libcfs] [c0000003ee34ea00] [8000000004533560] .osc_enter_cache+0x880/0x12c0 [osc] [c0000003ee34ebd0] [800000000453b210] .osc_queue_async_io+0xd10/0x1a40 [osc] [c0000003ee34edf0] [8000000004516fe8] .osc_page_cache_add+0xf8/0x2a0 [osc] [c0000003ee34eeb0] [80000000024468b8] .cl_page_cache_add+0xe8/0x3b0 [obdclass] [c0000003ee34efe0] [8000000004facae8] .lov_page_cache_add+0xc8/0x340 [lov] [c0000003ee34f0b0] [80000000024468b8] .cl_page_cache_add+0xe8/0x3b0 [obdclass] [c0000003ee34f1e0] [80000000067ecda4] .vvp_io_commit_write+0x474/0x8a0 [lustre] [c0000003ee34f300] [800000000246098c] .cl_io_commit_write+0x11c/0x2d0 [obdclass] [c0000003ee34f3c0] [80000000067af410] .ll_commit_write+0x120/0x3e0 [lustre] [c0000003ee34f490] [80000000067d1634] .ll_write_end+0x34/0x80 [lustre] [c0000003ee34f520] [c000000000098bac] .generic_file_buffered_write+0x1ec/0x374 [c0000003ee34f660] [c000000000099290] .__generic_file_aio_write+0x374/0x3d8 [c0000003ee34f760] [c00000000009936c] .generic_file_aio_write+0x78/0xe8 [c0000003ee34f810] [80000000067eff9c] .vvp_io_write_start+0xfc/0x3e0 [lustre] [c0000003ee34f8e0] [800000000245aedc] .cl_io_start+0xcc/0x220 [obdclass] [c0000003ee34f980] [8000000002462cf4] .cl_io_loop+0x194/0x2c0 [obdclass] [c0000003ee34fa30] [800000000676a278] .ll_file_io_generic+0x498/0x670 [lustre] [c0000003ee34fb30] [800000000676a8d4] .ll_file_aio_write+0x1d4/0x3a0 [lustre] [c0000003ee34fc00] [800000000676abf0] .ll_file_write+0x150/0x320 [lustre] [c0000003ee34fce0] [c0000000000d5968] .vfs_write+0xd0/0x1c4 [c0000003ee34fd80] [c0000000000d5b58] .SyS_write+0x54/0x98 [c0000003ee34fe30] [c000000000000580] syscall_exit+0x0/0x2c </pre> </div></div> <p>The specific line it's stuck at is here:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(gdb) l *osc_enter_cache+0x880 0x47f50 is in osc_enter_cache (/builddir/build/BUILD/lustre-2.3.56/lustre/osc/osc_cache.c:1526). 1521 /builddir/build/BUILD/lustre-2.3.56/lustre/osc/osc_cache.c: No such file or directory. in /builddir/build/BUILD/lustre-2.3.56/lustre/osc/osc_cache.c </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1523 CDEBUG(D_CACHE, "%s: sleeping for cache space @ %p for %p\n", 1524 cli-&gt;cl_import-&gt;imp_obd-&gt;obd_name, &amp;ocw, oap); 1525 1526 rc = l_wait_event(ocw.ocw_waitq, 1527 cfs_list_empty(&amp;ocw.ocw_entry), &amp;lwi); 1528 1529 client_obd_list_lock(&amp;cli-&gt;cl_loi_list_lock); 1530 cfs_list_del_init(&amp;ocw.ocw_entry); </pre> </div></div> <p>There were specific OSCs on a couple of the problem clients showing outstanding dirty pages for a significant amount of time:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>seqlac2@root:pdsh -w vulcanio[42-43] 'grep -v 0 /proc/fs/lustre/osc/*/cur_dirty_bytes' | dshbak -c ---------------- vulcanio42 ---------------- /proc/fs/lustre/osc/ls1-OST0026-osc-c0000003c1100000/cur_dirty_bytes:65536 /proc/fs/lustre/osc/ls1-OST0027-osc-c0000003c1100000/cur_dirty_bytes:65536 /proc/fs/lustre/osc/ls1-OST0028-osc-c0000003c1100000/cur_dirty_bytes:65536 /proc/fs/lustre/osc/ls1-OST002b-osc-c0000003c1100000/cur_dirty_bytes:65536 ---------------- vulcanio43 ---------------- /proc/fs/lustre/osc/ls1-OST0053-osc-c0000003e336b000/cur_dirty_bytes:65536 </pre> </div></div> <p>Manually dropping caches on the problem clients seems to have cleared things up:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># echo 3 &gt; /proc/sys/vm/drop_caches </pre> </div></div> <p>In case it is useful, attached is a dump of the lustre pages taken prior to dropping caches on one of the problem clients:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># cat /proc/fs/lustre/llite/ls1-c0000003e336b000/dump_page_cache &gt; vulcanio43-dump_page_cache-`date +%s` </pre> </div></div> <p>I also ran the same command after dropping caches, but the file was empty, so there's no reason to post it.</p> LU-2576

Hangs in osc_enter_cache due to dirty pages not being flushed

Bug Blocker Resolved Fixed Niu Yawei Prakash Surya MB ptr sequoia topsequoia Fri, 4 Jan 2013 18:37:48 +0000 Sat, 14 Sep 2013 17:28:46 +0000 Fri, 15 Feb 2013 21:33:29 +0000 Lustre 2.4.0 Lustre 2.4.0 0 9 <p>I should note we're running <a href="https://github.com/chaos/lustre/commits/2.3.57-2chaos" class="external-link" target="_blank" rel="nofollow noopener">2.3.57-2chaos</a> on these clients.</p> <p>Niu</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>In osc_enter_cache():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> rc = l_wait_event(ocw.ocw_waitq, cfs_list_empty(&amp;ocw.ocw_entry), &amp;lwi); </pre> </div></div> <p>I think we should take cl_loi_list_lock when checking the cfs_list_empty(&amp;ocw.ocw_entry), otherwise, the wakeup could be missed.</p> <p><a href="http://review.whamcloud.com/4963" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4963</a></p> <p>Did you do any sanity testing on 4963? We have testing time tomorrow on Sequoia, and I'm willing to pull it in if you have done some minimal testing. Otherwise I would wait for it to run through autotest and maybe test it on Thursday.</p> <p>Apparently this problem is that a dirty page couldn't be written out so that there is no I/O RPC between client and OST, this caused the writing process kept waiting for grants.</p> <p>I don't know why the dirty page was not written back. What are the value of /proc/sys/vm/dirty_XXX on the client node?</p> <p>Chris, please hold on. On second thought, I realized that 4963 might not be the real fix. Thanks.</p> <p>I think the wakeup condition in osc_enter_cache() is problematic: It checks only if the ocw is removed from the waiting list, however, in the wakup function osc_wake_cache_waiters(), it's possible return without removing the ocw if the global obd_dirty_pages is too high.</p> <p>I think we'd changed the wakup condition as before: list_empty(&amp;ocw-&gt;ocw_entry) || rpcs_in_flight(cli) == 0, then if the obd_dirty_pages is too high &amp; the dirty pages of this osc has been flushed, osc_enter_cache() will not go to sleep.</p> <p>not sure I understand, if there is already too much dirty_pages, it doesn't help anyway</p> <blockquote> <p>not sure I understand, if there is already too much dirty_pages, it doesn't help anyway</p></blockquote> <p>If there are too many dirty pages (for the whole client but not this specific osc, obd_dirty_pages &gt; obd_max_dirty_pages), osc_enter_cache() should return -EDQUOT to make this osc to start sync write, however with current code, osc_enter_cache() will go to sleep. (see osc_wake_cache_waiters(), no wakeup if obd_dirty_pages is too high)</p> <blockquote> <p>If there are too many dirty pages (for the whole client but not this specific osc, obd_dirty_pages &gt; obd_max_dirty_pages), osc_enter_cache() should return -EDQUOT to make this osc to start sync write</p></blockquote> <p>I don't see how the updated patch will cause <tt>osc_enter_cache()</tt> to return <tt>-EDQUOT</tt> in this case. If <tt>obd_dirty_pages &gt; obd_max_dirty_pages</tt> then <tt>ocw.ocw_rc</tt> will not get set to <tt>-EDQUOT</tt>, right? So then, how would <tt>osc_enter_cache()</tt> return <tt>-EDQUOT</tt>?</p> <blockquote> <p>I don't see how the updated patch will cause osc_enter_cache() to return -EDQUOT in this case. If obd_dirty_pages &gt; obd_max_dirty_pages then ocw.ocw_rc will not get set to -EDQUOT, right? So then, how would osc_enter_cache() return -EDQUOT?</p></blockquote> <p>I think you are right, we shouldn't override rc by ocw_rc in such case.</p> <p>Revision 5 looks like it might fix the issue. I'll pull it in to our branch for testing once it goes through Maloo without issues.</p> <blockquote> <p>I don't know why the dirty page was not written back. What are the value of /proc/sys/vm/dirty_XXX on the client node?</p></blockquote> <p>I haven't had access to a system hung in this state, but here's the values from one of the nodes:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ grep . /proc/sys/vm/dirty_* /proc/sys/vm/dirty_background_bytes:0 /proc/sys/vm/dirty_background_ratio:10 /proc/sys/vm/dirty_bytes:0 /proc/sys/vm/dirty_expire_centisecs:3000 /proc/sys/vm/dirty_ratio:20 /proc/sys/vm/dirty_writeback_centisecs:500 </pre> </div></div> <p>Seems the patchset 5 reveals another problem:</p> <p>In osc_enter_cache(), it's going to wait for grant when cli-&gt;cl_dirty &gt; 0, however osc_io_unplug() doesn't tirgger any IO because of oo_nr_writes == 0. I think there should be something wrong in the new extent code. Jingshan, any idea?</p> <p>It can be reproduced by replay-single test_88.</p> <blockquote> <p>In osc_enter_cache(), it's going to wait for grant when cli-&gt;cl_dirty &gt; 0, however osc_io_unplug() doesn't tirgger any IO because of oo_nr_writes == 0. I think there should be something wrong in the new extent code. Jingshan, any idea?</p></blockquote> <p>This is possible. If an extent is in OES_ACTIVE state, cl_dirty will be greater than 0 but oo_nr_writes can be 0. However, when the IO is finished, osc_extent_release() will detect that someone is waiting for grant and then unplug the queue immediately.</p> <p>I see the problem, in osc_enter_cache():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">while</span> (cli-&gt;cl_dirty &gt; 0 || cli-&gt;cl_w_in_flight &gt; 0) { osc_io_unplug(); wait_for_grant(); } </pre> </div></div> <p>Actually we shouldn't use 'while' here, because osc_io_unplug() can't trigger dirty flush for the object which isn't on the cl_loi_ready_list or cl_loi_hp_ready_list list (they do have few dirty pages, but not optimal to write out yet), in such case, I think we'd simply return -EDQUOT but not wait in the loop until the dirty pages were flushed.</p> <p>I don't know what the root cause is but apparently the above comment is incorrect because the queue will be unplugged no matter how many dirty pages there are, if there is cache waiters.</p> <p>Xiong, indeed, my previous comment was wrong. The real problem is that we don't have any API to trigger dirty flush on all objects now, we were using osc_check_rpcs() for this purpose before, but now it's wrapped in osc_io_unplug0(), and unfortunately, what osc_io_unplug0() does is: If current object needs be flushed, flush all dirty object, otherwise, do nothing.</p> <p>So if the current object in osc_enter_cache() has no dirty pages (it's the first page to be put in cache), dirty flush will not be triggered by osc_io_unplug().</p> <p>I realized that passing NULL to osc_io_unplug() can trigger dirty flush on all objects, will cook a new fix.</p> <p>Prakash, does the patch fix your problem? Thanks.</p> <p>We don't know yet. Other bugs keep preventing reasonable testing of the <a href="https://jira.whamcloud.com/browse/LU-2576" title="Hangs in osc_enter_cache due to dirty pages not being flushed" class="issue-link" data-issue-key="LU-2576"><del>LU-2576</del></a> patch. I see it landed to master. My hope is to get a good test run in today and leave a version of lustre with the <a href="https://jira.whamcloud.com/browse/LU-2576" title="Hangs in osc_enter_cache due to dirty pages not being flushed" class="issue-link" data-issue-key="LU-2576"><del>LU-2576</del></a> patch installed for users to try and see if things improve.</p> <p>Testing for this issue went well yesterday. We'll see how the patch holds up to real users over the next week, but I am cautiously optimistic that the problem is fixed.</p> <p>Has the testing continued to be successful and we can close this ticket?</p> <p>Yes, so far no repeats of this bug.</p> <p>patch landed for 2.4</p> <p>Dear All,</p> <p>Has this patch (#7) been really landed for 2.4 and 2.4.1-RC2?<br/> <a href="http://review.whamcloud.com/#/c/4963/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/4963/</a></p> <p>We have observed the issue for 2.4.0 and 2.4.1-RC2 so looks like the problem is still there.</p> <p>Here's our process stack:<br/> Sep 14 13:40:12 n0269-g6l kernel: ost_check S 0000000000000005 0 10821 1 0x00000080<br/> Sep 14 13:40:12 n0269-g6l kernel: ffff880127051548 0000000000000086 0000000000000000 0000000000000000<br/> Sep 14 13:40:12 n0269-g6l kernel: 0000000000000000 0000000000000082 ffff880127051528 ffffffff8106327e<br/> Sep 14 13:40:12 n0269-g6l kernel: ffff88020d5a3af8 ffff880127051fd8 000000000000fb88 ffff88020d5a3af8<br/> Sep 14 13:40:12 n0269-g6l kernel: Call Trace:<br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffff8106327e&gt;&#93;</span> ? try_to_wake_up+0x24e/0x3e0<br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa03236fe&gt;&#93;</span> cfs_waitq_wait+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa08d09ea&gt;&#93;</span> osc_enter_cache+0x85a/0xb40 <span class="error">&#91;osc&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa0338917&gt;&#93;</span> ? cfs_hash_bd_lookup_intent+0x37/0x130 <span class="error">&#91;libcfs&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffff81063410&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa08d7fd1&gt;&#93;</span> osc_queue_async_io+0x1081/0x1c18 <span class="error">&#91;osc&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffff810573c5&gt;&#93;</span> ? select_idle_sibling+0x95/0x150<br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffff8121ba89&gt;&#93;</span> ? security_capable+0x29/0x30<br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffff8107baba&gt;&#93;</span> ? capable+0x2a/0x60<br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa04841c5&gt;&#93;</span> ? cl_page_slice_add+0x55/0x140 <span class="error">&#91;obdclass&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa08bd2ec&gt;&#93;</span> ? osc_page_init+0xec/0x890 <span class="error">&#91;osc&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa08be2e9&gt;&#93;</span> osc_page_cache_add+0xc9/0x1d0 <span class="error">&#91;osc&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa04848af&gt;&#93;</span> cl_page_cache_add+0x7f/0x2a0 <span class="error">&#91;obdclass&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa094e335&gt;&#93;</span> lov_page_cache_add+0x85/0x200 <span class="error">&#91;lov&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa04848af&gt;&#93;</span> cl_page_cache_add+0x7f/0x2a0 <span class="error">&#91;obdclass&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa04889fc&gt;&#93;</span> ? cl_page_find0+0x44c/0x810 <span class="error">&#91;obdclass&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa0481439&gt;&#93;</span> ? cl_env_hops_keycmp+0x19/0x70 <span class="error">&#91;obdclass&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa03389bd&gt;&#93;</span> ? cfs_hash_bd_lookup_intent+0xdd/0x130 <span class="error">&#91;libcfs&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa0a26225&gt;&#93;</span> vvp_io_commit_write+0x3e5/0x5b0 <span class="error">&#91;lustre&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa0493cbd&gt;&#93;</span> cl_io_commit_write+0xad/0x1d0 <span class="error">&#91;obdclass&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa09fb37e&gt;&#93;</span> ll_commit_write+0xee/0x320 <span class="error">&#91;lustre&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa0a13650&gt;&#93;</span> ll_write_end+0x30/0x60 <span class="error">&#91;lustre&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffff8111a81a&gt;&#93;</span> generic_file_buffered_write+0x18a/0x2e0<br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffff81075887&gt;&#93;</span> ? current_fs_time+0x27/0x30<br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffff8111c210&gt;&#93;</span> __generic_file_aio_write+0x260/0x490<br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffff8111c4c8&gt;&#93;</span> generic_file_aio_write+0x88/0x100<br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa0a283db&gt;&#93;</span> vvp_io_write_start+0xcb/0x2e0 <span class="error">&#91;lustre&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa049072a&gt;&#93;</span> cl_io_start+0x6a/0x140 <span class="error">&#91;obdclass&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa0494e64&gt;&#93;</span> cl_io_loop+0xb4/0x1b0 <span class="error">&#91;obdclass&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa09ce9e0&gt;&#93;</span> ll_file_io_generic+0x450/0x600 <span class="error">&#91;lustre&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa09cf922&gt;&#93;</span> ll_file_aio_write+0x142/0x2c0 <span class="error">&#91;lustre&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffffa09cfc0c&gt;&#93;</span> ll_file_write+0x16c/0x2a0 <span class="error">&#91;lustre&#93;</span><br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffff81181368&gt;&#93;</span> vfs_write+0xb8/0x1a0<br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffff81181c61&gt;&#93;</span> sys_write+0x51/0x90<br/> Sep 14 13:40:12 n0269-g6l kernel: <span class="error">&#91;&lt;ffffffff8100b072&gt;&#93;</span> system_call_fastpath+0x16/0x1b</p> <p>Unfortunately dropping caches doesn't unblock the process. <br/> Only way to get ost communication to work again is to deactivate and reactivate osc</p> <p><span class="error">&#91;root@n0269-g6l ~&#93;</span># echo 0 &gt; /proc/fs/lustre/osc/scratch-OST0003-osc-ffff8802126ba400/active <br/> <span class="error">&#91;root@n0269-g6l ~&#93;</span># echo 1 &gt; /proc/fs/lustre/osc/scratch-OST0003-osc-ffff8802126ba400/active</p> <p>Could you please have a look?</p> <p>In case more debugging info is needed please let us know. We still keep some <br/> clients hanging for debugging purposes.</p> <p>Best Regards<br/> &#8211;<br/> Lukasz Flis<br/> ACC Cyfronet</p> <p>Hi Lukasz,</p> <p>Do you have <a href="http://review.whamcloud.com/6554" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6554</a> in your tree? It looks like we need to cherry pick this patch into b2_4 as well.</p> <p>patch for b2_4 is at: <a href="http://review.whamcloud.com/7657" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7657</a></p> <p>Hi Jinshan,</p> <p>Thank you very much for pointing this one out and for patch for 2.4<br/> we'll give it a try ASAP. There was no sign of this change in 2.4.0 and 2.4.1RC2</p> <p>I see that 2.4.1 is ready for release (packages are ready and published) - maybe it's worth <br/> to consider holding (or updating) release a bit as this bug can manifest itself in few hours <br/> (i have an impression that oss recoveries raise the chance of getting this one)</p> <p>Thank you very much for the quick patch. I'll try to give a feedback asap</p> <p>&#8211;<br/> Lukasz Flis</p> Related LU-2139 LU-3261 LU-3416 LU-3277 LU-2582 Development Rank 1|hzven3: Rank (Obsolete) 6013 Severity