https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2640 Lustre <p>In <a href="http://review.whamcloud.com/5046" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5046</a> there was a change to <tt>OSD_EXEC_OP()</tt> to address problems in MDD operations that fail part-way through, where the undo of earlier operations causes internal osd-ldiskfs declare/execute accounting to fail. For example, in <tt>mdd_create()::cleanup</tt>, the failed create calls <tt>__mdd_index_delete(), mdo_ref_del(), mdo_destroy()</tt> in order to clean up the newly created object.</p> <p>One proposal is to have some kind of OSD API call/flag/method to mark the transaction handle as being used for rollback, and to disable the ot_declare_op LASSERT() checking in OSD_EXEC_OP() for this case.</p> LU-2640

deactivate OSD_EXEC_OP() operation accounting if operation is being undone

Bug Blocker Resolved Fixed nasf Andreas Dilger MB Fri, 18 Jan 2013 00:14:24 +0000 Wed, 13 Sep 2017 06:34:33 +0000 Mon, 18 Mar 2013 23:18:11 +0000 Lustre 2.4.0 Lustre 2.4.0 0 6 <p>To clarify, this bug is to track this fix separately from the LFSCK 1.5 project, instead of submitting the fix merged into that patch series.</p> <p>given all this information is required only from create to stop (and not longer), we can reserve quite amount of memory in osd_thread_info to track per-inode declaration. this way we could avoid the need in additional method.</p> <p>Currently, I think that changing OSD APIs to enhance transaction debug for tracing modification rollback is some overreaction: not only the declare APIs, but also some exec APIs, otherwise, the OSD cannot distinguish whether a modification, such as destroy, belongs to a normal unlink or create rollback.</p> <p>So I prefer to simplify the requirement and processing as that:</p> <p>1) When declare for the modification, the transaction trace mechanism will record both the transaction itself and its possible rollback. For example, when declare for ref_add, both "osd_thandle::ot_declare_ops<span class="error">&#91;OSD_OT_REF_ADD&#93;</span>" and "osd_thandle::ot_declare_ops_rb<span class="error">&#91;OSD_OT_REF_ADD&#93;</span>" will be increased.</p> <p>2) The OSD needs not to strictly distinguish whether a modification belongs to a normal transaction or a rollback. For example, when ref_del is called, it will try to check "osd_thandle::ot_declare_ops<span class="error">&#91;OSD_OT_REF_DEL&#93;</span>" firstly, if failed to decrease, then it quite possible belongs to former ref_add rollback, so the "osd_thandle::ot_declare_ops_rb<span class="error">&#91;OSD_OT_REF_ADD&#93;</span>" will be decreased.</p> <p>Although it is not perfect, it is enough to resolve the issues I met during LFSCK 1.5 test. I prefer to use this patch (<a href="http://review.whamcloud.com/#change,5138" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5138</a>) as a solution, at least a temporary solution for <a href="https://jira.whamcloud.com/browse/LU-2640" title="deactivate OSD_EXEC_OP() operation accounting if operation is being undone" class="issue-link" data-issue-key="LU-2640"><del>LU-2640</del></a>.</p> <p>It does not belong to the LFSCK, and not depends on the LFSCK. But some sanity tests may fail if without this issue fixed, it should be landed to master in priority.</p> <p>well, I think Andreas is expecting a bit more than just fixing this assertion (otherwise we could just disable the whole thing which is just silly sanity check at the moment). instead we should think how we can improve credits accounting as well. I think this can be done on per-inode/op basis.</p> <p>Well, it is different from transaction debug disable, it still has original functionality, additional to allow rollback cases. About more accurate credits accounting, we need consider more, but it seems not urgent requirement.</p> <p>To not block sanity tests, can we consider some temporary and simple solution?</p> <p>not sure I follow. we do pass sanity tests, there might be some cases, but they do not block testing. </p> <p>We passed sanity test, just because we did not correctly rollback something. Please check mdd_dir.c changes in the patch <a href="http://review.whamcloud.com/#change,5138" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5138</a></p> <p>this doesn't sound like a blocker for testing, because 5138 itself can be postponed. for release, sure.</p> <p>The issues fixed in mdd_dir.c will affect LFSCK behavior. If we do not fix it, then there may be some dangling reference name entries in the system.</p> <p>well, i didn't say "do not fix it". but i think there is no point to introduce some temporary solution because for a while it's enough to just disable this sanity check.<br/> and AFAIU, this "dangling reference" should not block the testing.</p> <p>The "dangling reference" will not affect sanity test, but it will affect the LFSCK behavior.<br/> I prefer to well define the policy for name entry insert/delete, and the target object nlink add/del. The current implementation is confused and contains bugs.</p> <p>not sure what "policy" means here. the exact order should not matter much. if current implementation is missing some undo, then it's to be fixed, of course.<br/> as for LFSCK, it should be able to cope with this as there are already filesystems using this code, so having dangling reference in theory.</p> <p>Yes, in theory, for existing system, the dandling reference name entry may exist. But we should try to avoid that in new system.</p> <p>As my understand, we prefer to space lost rather than dangling reference, which has been explained in mdd_create() comment. I think we should follow the similar policy for link/unlink cases.</p> <p>we sort of having transactions to avoid this.. if some change can't be undone, then i'd suppose all the processing should be aborted (read-only and failover in case of ldiskfs). until the change is committed with -&gt;trans_stop() on-disk state is safe (or not worse than before -&gt;trans_start())</p> <p>I have updated the patch (<a href="http://review.whamcloud.com/#change,5138" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5138</a>) with more consideration.</p> <p>Hi Nasf, I have checked the latest patch and got fail, pls look at <a href="http://jira.whamcloud.com/browse/LU-2668?focusedCommentId=51027&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-51027" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-2668?focusedCommentId=51027&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-51027</a></p> <p>Miss to call "OSD_EXEC_OP(th, insert)" in "osd_index_ea_insert()". It exists in master for a long time. Thanks for trigger it!</p> <p>I will update my patches to fix it.</p> <p>Alexander, what are you using for testing this problem? It looks like the MDT is running out of space and fails in the error cleanup path.</p> <p>Andreas,</p> <p>Alexander worked on backport a directory size limits patch from upstream to lustre to avoid problems from <a href="https://jira.whamcloud.com/browse/LU-2479" title="sanity.sh test_129: max dir size limit work incorrectly." class="issue-link" data-issue-key="LU-2479"><del>LU-2479</del></a>.<br/> quickly - we have created large directory, remove some entries, set limit, but not able to create something because that directory over limits. patch mainline, just forbid to extent a directory instead of creating anything.</p> <p>I try patchset 3</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00000001:1.0:1359025663.382680:0:14733:0:(mdd_dir.c:523:__mdd_index_insert_only()) Process leaving (rc=18446744073709551588 : -28 : ffffffffffffffe4) 00000004:00000001:1.0:1359025663.382681:0:14733:0:(mdd_dir.c:540:__mdd_index_insert()) Process leaving (rc=18446744073709551588 : -28 : ffffffffffffffe4) 00000004:00000001:1.0:1359025663.382682:0:14733:0:(mdd_dir.c:1794:mdd_create()) Process leaving via cleanup (rc=18446744073709551588 : -28 : 0xffffffffffffffe4) 00000004:00040000:1.0:1359025663.382684:0:14733:0:(osd_internal.h:412:osd_trans_exec_op()) ASSERTION( oh-&gt;ot_declare_ops_rb[rb] &gt; 0 ) failed: 5 00000004:00040000:1.0:1359025663.382799:0:14733:0:(osd_internal.h:412:osd_trans_exec_op()) LBUG Pid: 14733, comm: mdt01_001 Call Trace: [&lt;ffffffffa073e905&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa073ef07&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0e7d6e0&gt;] osd_it_ea_get+0x0/0x140 [osd_ldiskfs] [&lt;ffffffffa0e86c75&gt;] osd_object_ref_del+0x135/0x210 [osd_ldiskfs] [&lt;ffffffffa0bb132b&gt;] lod_ref_del+0x3b/0xd0 [lod] [&lt;ffffffffa0dbf5fd&gt;] mdo_ref_del+0xad/0xb0 [mdd] [&lt;ffffffffa0dc5ce2&gt;] mdd_create+0xd02/0x1500 [mdd] [&lt;ffffffffa074f371&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa0f27321&gt;] mdt_reint_open+0x1191/0x1900 [mdt] [&lt;ffffffffa074f371&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa0f121a1&gt;] mdt_reint_rec+0x41/0xe0 [mdt] [&lt;ffffffffa0f0b843&gt;] mdt_reint_internal+0x4e3/0x7d0 [mdt] [&lt;ffffffffa0f0bdfd&gt;] mdt_intent_reint+0x1ed/0x4f0 [mdt] [&lt;ffffffffa0f074fe&gt;] mdt_intent_policy+0x3ae/0x750 [mdt] </pre> </div></div> <p>mdd_declare_create() has one declare_ref_add, but mdd_create() cleanup has two ref_del().</p> <p>&gt; mdd_declare_create() has one declare_ref_add, but mdd_create() cleanup has two ref_del().</p> <p>right, because mdd_create() does not declare initial nlink set to 1 by object creation<br/> (we discussed this problem few times in the past..)<br/> adding one more declare_ref_add() in mdd_create() w/o actual ref_add() would be confusing, imho.</p> <p>Seems we do not need two ref_del() when rollback for mkdir, the osd_object_destroy() can handle the last nlink for dir object.</p> <p>Alex, how do you think?</p> <p>well, this is not quite correct. osd_object_destroy() does its own hack because of limits on nlink in ldiskfs. we should not bring this assumption into the API.</p> <p>Yes, it's hack. Then what do you prefer to resolve it?</p> <p>We can hack in OSD, and let osd_thandle::ot_declare_ops_rb<span class="error">&#91;OSD_OT_REF_ADD&#93;</span> increase when osd_trans_exec_op(OSD_OT_CREATE) called, can you accept it?</p> <p>of course, it's better. though I was actually hoping for a breakthrough solution to improve checks and credits calculation.</p> <p>Alexander, would you please to try this one? Thanks.</p> <p><a href="http://review.whamcloud.com/#change,5138,set4" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5138,set4</a></p> <p>With patch set 4 test looks fine.</p> <p>Alex, the 5138 patch was landed as-is to allow later LFSCK patch landings to move forward.</p> <p>Oleg wants a better comment explaining what the transaction declaration accounting is doing, so that it can be more easily understood by others.</p> <p>Other open questions that need to be discussed:</p> <ul class="alternate" type="square"> <li>should we implement an OSD API to notify the transaction that there is a rollback in progress?</li> <li>having more accurate transaction accounting is useful for both reducing the transaction credits, and improving the debugging. We do some limited aggregation of declarations for quotas, so that the common case of multiple updates for the same UID/GID do not need too many credits. We could change this to be a more generic mechanism (e.g. identified by FID?) that tracks a few different kinds of updates (e.g. setattr/dirty object, write/append/alloc/punch/truncate, insert/delete item).</li> </ul> <p>if I remember DMU code correctly, it's fine to declare some range to be modified and then modify it few times - we should be passing internal sanity checks.<br/> the same applies for ldiskfs for sure.<br/> given so I think that we don't actually need to declare "rollback" updates as they are working on the same range/hash always.<br/> notice we probably don't need this mechanism with DMU as it does this internally (when debugging is enabled).</p> <p>One idea about the "rollback" notification: introduce a new member thandle::th_rollback, which can be set by MDD and checked by OSD.</p> <p>How do you think?</p> <p>the thing is that neither ldiskfs nor dmu need this.. </p> <p>That depends on how "need" to be defined. From the view of transaction tracing and debugging, it is useful. We may implement such purposes through other ways, but the "thandle::th_rollback" is simple and clean. In fact, the "thandle::th_local" is also used for notification among layers, but not the functional requirement of ldiskfs or DUM. So...</p> <p>this debugging is in turn needed only because low-level debugging in ldiskfs is rather poor. this is the only reason to have it in the first place.<br/> with better debugging (which we want to calculate credits better) we likely won't need this flag.</p> <p>Alex, while I agree that th_rollback is not strictly needed, I think you two are arguing in circles. Clearly we want to improve the ldiskfs transaction credit debugging in some way. The current mechanism is rather weak in detecting when rollback has started, while setting a flag in the handle is positively indicating that rollback has started. </p> <p>Unless you have some better mechanism that we could use in short order, I'm inclined to go with this method for now. The only other method I can think of is adding detailed accounting for every item being modified (e.g. tracking inode numbers, group descriptors, bitmaps, etc) so that the same item can be modified multiple times. However the complexity of that is high and I think it could be deferred until 2.5 if we want to implement it at that point.</p> <p>For now, I think th_rollback is an acceptable solution for the time we have left in 2.4, and it can be clearly commented that it is a workaround and reference this bug for a more complete solution in the future.</p> <p>it doesn't need to be that detailed, imo. just an array of object and (depending on object type) optype with some argument (hash in case of index, range in case of regular file, xattr) would be enough.</p> <p>I am against th_rollback because we'll have to modify number of functions in MDD to implement this. and then this will stay there for long as no one seem to be interested enough.</p> <p>at the same time 2.x is doing pretty well w/o this mechanism at all (obviously). I'm tempted to say i regret for introducing this in the first place <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>given we do not hit this (except undo cases) so regular paths should be fine from this point of view, I'd suggest to disable the mechanism.<br/> we still have ldiskfs's internal checks.</p> <p>Do you think the current mechanism is worse than none at all? I think the current accounting can still find some code defects, even if it does not find as many as with th_rollback being set by the caller. In that case, I'd rather leave it in place during development instead of removing it entirely.</p> <p>Maybe make this debugging conditional on (LUSTRE_PATCH &gt;= 50 &amp;&amp; LUSTRE_PATCH &lt; 90)? That allows us to catch problems during development, but relies only on ldiskfs/jbd accounting during production. </p> <p>well, if it doesn't work well on its own, i'd rather agree it's bad. in orion i made the most trivial implementation just to avoid some silly bugs and frankly i didn't meant to keep it this way for long. unfortunately it survived.</p> <p>i'm absolutely fine with your idea to have it enabled in pre-release versions or something like that. i'm not actually going to block the patch if you think this is required. but again my concern is that we keep filling generic code with more and more very specific handling (another example is fid_is_client_mdt_visible())<br/> which shouldn't be there at all.</p> <p>The patch for thandle::th_rollback idea:</p> <p><a href="http://review.whamcloud.com/#change,5501" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5501</a></p> <p>The patch for conditional enable transaction debug:</p> <p><a href="http://review.whamcloud.com/#change,5698" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5698</a></p> <p>Landed for 2.4</p> Duplicate LU-2668 Development Rank 1|hzvflr: Rank (Obsolete) 6174 Severity