https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16044 Lustre <blockquote> <p>PID: 12333 TASK: ffff8d84a294c200 CPU: 8 COMMAND: "ll_ost_io02_086"<br/> #0 <span class="error">&#91;ffff8d84a29937e0&#93;</span> __schedule at ffffffffa8988e18<br/> #1 <span class="error">&#91;ffff8d84a2993848&#93;</span> schedule at ffffffffa89891e9<br/> #2 <span class="error">&#91;ffff8d84a2993858&#93;</span> schedule_timeout at ffffffffa8986eb1<br/> #3 <span class="error">&#91;ffff8d84a2993908&#93;</span> io_schedule_timeout at ffffffffa8988a9d<br/> #4 <span class="error">&#91;ffff8d84a2993938&#93;</span> io_schedule at ffffffffa8988b38<br/> #5 <span class="error">&#91;ffff8d84a2993948&#93;</span> bit_wait_io at ffffffffa8987501<br/> #6 <span class="error">&#91;ffff8d84a2993960&#93;</span> __wait_on_bit_lock at ffffffffa89870b1<br/> #7 <span class="error">&#91;ffff8d84a29939a0&#93;</span> __lock_page at ffffffffa83bd2a4<br/> #8 <span class="error">&#91;ffff8d84a29939f8&#93;</span> truncate_inode_pages_range at ffffffffa83cf2fb<br/> #9 <span class="error">&#91;ffff8d84a2993b50&#93;</span> truncate_pagecache at ffffffffa83cf3f7<br/> #10 <span class="error">&#91;ffff8d84a2993b78&#93;</span> osd_punch at ffffffffc14beecc <span class="error">&#91;osd_ldiskfs&#93;</span><br/> #11 <span class="error">&#91;ffff8d84a2993bd0&#93;</span> ofd_object_punch at ffffffffc15e7e26 <span class="error">&#91;ofd&#93;</span><br/> #12 <span class="error">&#91;ffff8d84a2993c48&#93;</span> ofd_punch_hdl at ffffffffc15d442f <span class="error">&#91;ofd&#93;</span><br/> #13 <span class="error">&#91;ffff8d84a2993cd0&#93;</span> tgt_checksum_niobuf_t10pi at ffffffffc0fe909e <span class="error">&#91;ptlrpc&#93;</span><br/> #14 <span class="error">&#91;ffff8d84a2993d58&#93;</span> ptlrpc_server_handle_request at ffffffffc0f9090b <span class="error">&#91;ptlrpc&#93;</span><br/> #15 <span class="error">&#91;ffff8d84a2993df8&#93;</span> ptlrpc_main at ffffffffc0f94274 <span class="error">&#91;ptlrpc&#93;</span><br/> #16 <span class="error">&#91;ffff8d84a2993ec8&#93;</span> kthread at ffffffffa82c5e31<br/> ......</p> <p>PID: 12603 TASK: ffff8d8490db0000 CPU: 14 COMMAND: "ll_ost_io05_068"<br/> #0 <span class="error">&#91;ffff8d8490daf8a8&#93;</span> __schedule at ffffffffa8988e18<br/> #1 <span class="error">&#91;ffff8d8490daf910&#93;</span> schedule at ffffffffa89891e9<br/> #2 <span class="error">&#91;ffff8d8490daf920&#93;</span> rwsem_down_read_failed at ffffffffa898abd5<br/> #3 <span class="error">&#91;ffff8d8490daf9a0&#93;</span> call_rwsem_down_read_failed at ffffffffa8598068<br/> #4 <span class="error">&#91;ffff8d8490daf9f0&#93;</span> down_read at ffffffffa89886b0<br/> #5 <span class="error">&#91;ffff8d8490dafa08&#93;</span> osd_read_lock at ffffffffc148e03c <span class="error">&#91;osd_ldiskfs&#93;</span><br/> #6 <span class="error">&#91;ffff8d8490dafa30&#93;</span> ofd_commitrw_write at ffffffffc15eb76c <span class="error">&#91;ofd&#93;</span><br/> #7 <span class="error">&#91;ffff8d8490dafac0&#93;</span> ofd_commitrw at ffffffffc15efe4f <span class="error">&#91;ofd&#93;</span><br/> #8 <span class="error">&#91;ffff8d8490dafb58&#93;</span> tgt_request_preprocess at ffffffffc0fee11b <span class="error">&#91;ptlrpc&#93;</span><br/> #9 <span class="error">&#91;ffff8d8490dafcd0&#93;</span> tgt_checksum_niobuf_t10pi at ffffffffc0fe909e <span class="error">&#91;ptlrpc&#93;</span><br/> #10 <span class="error">&#91;ffff8d8490dafd58&#93;</span> ptlrpc_server_handle_request at ffffffffc0f9090b <span class="error">&#91;ptlrpc&#93;</span><br/> #11 <span class="error">&#91;ffff8d8490dafdf8&#93;</span> ptlrpc_main at ffffffffc0f94274 <span class="error">&#91;ptlrpc&#93;</span><br/> #12 <span class="error">&#91;ffff8d8490dafec8&#93;</span> kthread at ffffffffa82c5e31</p></blockquote> LU-16044

osd: truncate vs write deadlock

Bug Minor Resolved Fixed Alex Zhuravlev Alex Zhuravlev Mon, 25 Jul 2022 13:25:49 +0000 Mon, 19 Jun 2023 12:09:24 +0000 Sun, 16 Oct 2022 01:05:18 +0000 Lustre 2.16.0 0 5 <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/48033" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48033</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16044" title="osd: truncate vs write deadlock" class="issue-link" data-issue-key="LU-16044"><del>LU-16044</del></a> osd: discard pagecache in truncate's declaration<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f7f107f3f0144b5f15309506dc6bc3509c1d8d70</p> <p>"Stephane Thiell &lt;sthiell@stanford.edu&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/48410" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48410</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16044" title="osd: truncate vs write deadlock" class="issue-link" data-issue-key="LU-16044"><del>LU-16044</del></a> osd: discard pagecache in truncate's declaration<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: de4c30e20f4d474ec363835f2ce2456d23896cc4</p> <p>Alex, this is a backport of your patch to b2_12. Basically I just removed the encryption part that is not available in 2.12. Can you please double check this looks OK to you? When I get your go, we'll try it in production. Thanks!</p> <p>Unfortunately, even with 2.12.9 with both patches:</p> <ul> <li><a href="https://jira.whamcloud.com/browse/LU-16044" title="osd: truncate vs write deadlock" class="issue-link" data-issue-key="LU-16044"><del>LU-16044</del></a> osd: discard pagecache in truncate's declaration (<a href="https://review.whamcloud.com/48410" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48410</a></li> <li><a href="https://jira.whamcloud.com/browse/LU-15117" title="ofd_read_lock vs transaction deadlock while allocating buffers " class="issue-link" data-issue-key="LU-15117"><del>LU-15117</del></a> ofd: don't take lock for dt_bufs_get() (<a href="https://review.whamcloud.com/47925," class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47925)</a></li> </ul> <p>We hit a deadlock situation last night. Attaching "foreach bt" of the new crash dump as <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/45632/45632_foreach_bt_fir-io2-s2_20220909.txt" title="foreach_bt_fir-io2-s2_20220909.txt attached to LU-16044">foreach_bt_fir-io2-s2_20220909.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>We may have identified the source of the deadlock. A group of users had jobs using GNU parallel with --tmpfile set to Lustre, which apparently uses unlinked tmp files that are kept opened and it does frequent ftruncate(0) on them.</p> <p>The command used is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>parallel --tmpdir $folder/tmp --delay 2 -j $threads &lt; $folder/calls.$cmd.txt </pre> </div></div> <p>with <tt>$folder</tt> set to Lustre</p> <p>We have asked the users to change their scripts and avoid Lustre to store such temporary files, and we'll see if that reduces the number of OSS deadlock.</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> schedule,io_schedule,bit_wait_io,__wait_on_bit_lock,__lock_page,mpage_prepare_extent_to_map,ldiskfs_writepages,do_writepages,__writeback_single_inode,writeback_sb_inodes,__writeback_inodes_wb,wb_writeback,bdi_writeback_workfn,process_one_work,worker_thread PIDs(1): <span class="code-quote">"kworker/u259:2"</span>:94708 </pre> </div></div> <p>this is not the problem I tried to fix. probably better to say it's a related issue. need to think a bit more.. sorry for the inconvenience.</p> <blockquote><p>We have asked the users to change their scripts and avoid Lustre to store such temporary files, and we'll see if that reduces the number of OSS deadlock.</p></blockquote> <p>please ask for sysctl -a | grep vm.dirty from OSTs</p> <p>Thanks Alex. This is the result from all our OSS on this system. I believe we use the default settings for CentOS 7.9.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>--------------- fir-io[1-8]-s[1-2] (16) --------------- vm.dirty_background_bytes = 0 vm.dirty_background_ratio = 3 vm.dirty_bytes = 0 vm.dirty_expire_centisecs = 3000 vm.dirty_ratio = 10 vm.dirty_writeback_centisecs = 500 </pre> </div></div> <p>What we change is the following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>vm.min_free_kbytes = 2097152 vm.swappiness = 1 vm.zone_reclaim_mode = 1 </pre> </div></div> <p>OSS are based on AMD EPYC Naples, single socket 7401P. 512GB of RAM each.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48033/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48033/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16044" title="osd: truncate vs write deadlock" class="issue-link" data-issue-key="LU-16044"><del>LU-16044</del></a> osd: discard pagecache in truncate's declaration<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0bb491b2ecf494c3f78fa08a101af8af7853a0fe</p> <p>Landed for 2.16</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51360" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51360</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16044" title="osd: truncate vs write deadlock" class="issue-link" data-issue-key="LU-16044"><del>LU-16044</del></a> osd: discard pagecache in truncate's declaration<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 536d362534f37e53bae1868b4ea1a044306b69a4</p> Related Development Rank 1|i02ve7: Rank (Obsolete) 9223372036854775807 Severity