https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10048 Lustre <p>this is needed to implement (transaction first; locking next) order to unify locking among MDT/OST/OUT</p> LU-10048

osd-ldiskfs to truncate outside of main transaction

Bug Major Resolved Fixed Alex Zhuravlev Alex Zhuravlev Fri, 29 Sep 2017 10:11:57 +0000 Thu, 7 Jul 2022 22:11:19 +0000 Tue, 27 Aug 2019 17:54:03 +0000 Lustre 2.12.0 Lustre 2.10.4 Lustre 2.10.5 Lustre 2.13.0 0 19 <p>The <a href="https://review.whamcloud.com/27488" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27488</a> patch is for ldiskfs, while the <a href="https://jira.whamcloud.com/browse/LU-8806" title="LFSCK hangs on MDT - osp_precreate_cleanup_orphans - cannot cleanup orphans: rc = -107" class="issue-link" data-issue-key="LU-8806"><del>LU-8806</del></a> patch is for ZFS. Presumably they both need to be fixed?</p> <p>Alex Zhuravlev (alexey.zhuravlev@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31293" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31293</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> ofd: take local locks within transaction<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a0ff9fb6374689e50674901f3ff3ede857d61848</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/27488/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27488/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> osd: async truncate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: cf29a5e7bfa254ccfcea023028fe7da80503c512</p> <p>Still one more patch to land. </p> <p>is there a backport for b2_10 available or planned?</p> <p>Alex pointed this issue as duplicate of <a href="https://jira.whamcloud.com/browse/LU-11465" title="OSS/MDS deadlock in 2.10.5" class="issue-link" data-issue-key="LU-11465"><del>LU-11465</del></a>  but we were not able to cherrypick changes in b2_10</p> <p>We are experiencing MDT/OST lock-ups on 2_10_5 few times a day in the worst case</p> <p> </p> <p> </p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33586" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33586</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> osd: async truncate<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 414b3ed9fd5aa975da26b8847e9cfe8a188b59ce</p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33587" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33587</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> ofd: take local locks within transaction<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 7b6cdb405a519b11537316956affc4bfa4467f8d</p> <p>Are the 2.10 ok to use? or do they still need additional work?</p> <p> </p> <p>Mahmoud</p> <p>I would recommend holding off for now</p> <p>Peter</p> <p>I think the <a href="https://review.whamcloud.com/33586" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33586</a> patch "<a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> osd: async truncate" is relatively safe - it has been in master for several months already, but I'm not sure it will fix the issue completely. Also, the patch <a href="https://review.whamcloud.com/33682" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33682</a><br/> "Revert <a href="https://jira.whamcloud.com/browse/LU-5152" title="Can&#39;t enforce block quota when unprivileged user change group" class="issue-link" data-issue-key="LU-5152"><del>LU-5152</del></a> quota: enforce block quota for chgrp" may also help with MDT/OST lockups. It reverts a patch that was landed in 2.10.4 that introduced a circular dependency between servers in the quota handling.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/31293/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31293/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> ofd: take local locks within transaction<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9f79d4488fbb466647d1d09c2e6a1d3555d062fc</p> <p>So...is this ok to mark as resolved now?</p> <p>Andreas thinks yes</p> <p>Mark Roper (markroper@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/37797" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37797</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> ofd: take local locks within transaction<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: afa505898952b9c094bd6fb4d9911032e4eefb68</p> <p>We submitted a backport of this patch to b2_10 if anybody is looking for it.</p> <p>@Lukasz, if you are still looking for it <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p> </p> <p>Etienne AUJAMES (eaujames@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/43277" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43277</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> ofd: take local locks within transaction<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 4dc5634dcdf3381a7ea698605a3c7a345f689a4c</p> <p>Hello,</p> <p>I have backported the <a href="https://review.whamcloud.com/43277" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43277</a> ("<a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> ofd: take local locks within transaction") along with the <a href="https://review.whamcloud.com/43278" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43278</a> ("<a href="https://jira.whamcloud.com/browse/LU-13093" title="osd_trans_exec_check()) lustre-OST0001: opcode 0: used 8, used now 8, reserved 1 LBUG" class="issue-link" data-issue-key="LU-13093"><del>LU-13093</del></a> osd: fix osd_attr_set race").</p> <p>It seems we trigger that bug in 2.12.6 with external journal (on flash dev) for rotational disk while several migrate thread is running on a robinhood client (512 thread).</p> <p>jbd2 journal seems to be locked when the transaction is in T_LOCKED.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> transaction_t.t_state = T_LOCKED transaction_t.t_handle_count = 40 transaction_t.t_updates = 1 nbr of task in j_wait_transaction_locked = 324 nbr of task in j_wait_update = 0 </pre> </div></div> <p>there is yet another patch under <a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> - <a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> osd: async truncate, cf29a5e7bfa254ccfcea023028fe7da80503c512 in master</p> <p>as for fast journal - we've got number of nodes running all the tests on RAM-backed devices 24h a day.</p> <p>The "<a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> osd: async truncate" has already landed on b2_12.</p> <p>please, generate full backtrace (</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">echo t &gt;/proc/sysrq-trigger</pre> </div></div> <p>) and attach to the ticket</p> <p>The issue seems to occurs relatively often with a lot of migrate threads (today 4 times on different OSTs).</p> <p>We will test the backports quickly on a production environment (maybe this week).</p> <p>I will try to get backtrace from the crashdump (manually triggered) tomorrow.</p> <p>I have added the our backtrace to this tickets: <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/38320/38320_crash_bt_jbd2_locked_20210412.log" title="crash_bt_jbd2_locked_20210412.log attached to LU-10048">crash_bt_jbd2_locked_20210412.log^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>Hello Alex,</p> <p>Did you have the time to look to our backtrace?</p> <p>If you need more data from the crashdump I can get you some.</p> <p>Hello,</p> <p>We have applied the <a href="https://review.whamcloud.com/43277" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43277</a> ("<a href="https://jira.whamcloud.com/browse/LU-10048" title="osd-ldiskfs to truncate outside of main transaction" class="issue-link" data-issue-key="LU-10048"><del>LU-10048</del></a> ofd: take local locks within transaction") + <a href="https://review.whamcloud.com/43278" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43278</a> ("<a href="https://jira.whamcloud.com/browse/LU-13093" title="osd_trans_exec_check()) lustre-OST0001: opcode 0: used 8, used now 8, reserved 1 LBUG" class="issue-link" data-issue-key="LU-13093"><del>LU-13093</del></a> osd: fix osd_attr_set race") on the problematic filesystem. The issue never occurred after.</p> <p>We were able to reproduce this issue in 5/10 min with many creations of small files and several threads doing file migrations ("lfs migrate" between OSTs).</p> Blocker LU-11764 Duplicate LU-11613 LU-8786 Related LU-5152 LU-8806 LU-11685 LU-11465 LU-5994 LU-13234 LU-12977 Development Rank 1|hzzl07: Rank (Obsolete) 9223372036854775807