https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6634 Lustre <p>I found this problem during my testing of my patch for <a href="https://jira.whamcloud.com/browse/LU-6556" title="changelog catalog corruption if all possible records is define " class="issue-link" data-issue-key="LU-6556"><del>LU-6556</del></a>, when also trying verify a Catalog full condition ends up with ENOSPC.</p> <p>Having a look to the code, this happens because in the error path for llog_cat_new_log(), llog_destroy() is called to destroy the new plain LLOG for which the reference can't be recorded into Catalog because there is no slot available to do so.<br/> Doing so triggers the Assertion because there is already a started transaction from llog_cat_add(), when llog_destroy() wants to start its own transaction.</p> LU-6634

(osd_handler.c:901:osd_trans_start()) ASSERTION( get_current()->journal_info == ((void *)0) ) failed: when reaching Catalog full condition

Bug Critical Resolved Fixed Alex Zhuravlev Bruno Faccini Sat, 23 May 2015 21:49:32 +0000 Fri, 18 Dec 2015 19:40:15 +0000 Wed, 4 Nov 2015 14:38:09 +0000 Lustre 2.7.0 Lustre 2.8.0 Lustre 2.8.0 0 10 <p>The full LBUG/Assert signature is :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 11337:0:(llog_osd.c:530:llog_osd_write_rec()) no free catalog slots for log... LustreError: 11337:0:(osd_handler.c:901:osd_trans_start()) ASSERTION( get_current()-&gt;journal_info == ((void *)0) ) failed: LustreError: 11337:0:(osd_handler.c:901:osd_trans_start()) LBUG Pid: 11337, comm: lctl Call Trace: [&lt;ffffffffa042f875&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa042fe77&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa12616cc&gt;] osd_trans_start+0x25c/0x660 [osd_ldiskfs] [&lt;ffffffffa054ef0a&gt;] llog_osd_destroy+0x42a/0xd40 [obdclass] [&lt;ffffffffa0548e6c&gt;] llog_cat_new_log+0x1ec/0x710 [obdclass] [&lt;ffffffffa054949a&gt;] llog_cat_add_rec+0x10a/0x450 [obdclass] [&lt;ffffffffa05499f5&gt;] llog_cat_add+0x215/0x380 [obdclass] [&lt;ffffffffa16c8905&gt;] llog_test_9+0xf15/0x11a0 [llog_test] [&lt;ffffffffa054a200&gt;] ? llog_ctxt_destroy+0x160/0x1f0 [obdclass] [&lt;ffffffffa16c55ff&gt;] ? verify_handle+0x1f/0x240 [llog_test] [&lt;ffffffffa16cde2a&gt;] llog_run_tests+0x165a/0x1a00 [llog_test] [&lt;ffffffffa16ce979&gt;] llog_test_setup+0x7a9/0x8f0 [llog_test] [&lt;ffffffffa05749cb&gt;] obd_setup+0x19b/0x290 [obdclass] [&lt;ffffffff8116fb23&gt;] ? kmem_cache_alloc_trace+0x1a3/0x1b0 [&lt;ffffffffa055f9db&gt;] ? class_new_export+0x78b/0x9a0 [obdclass] [&lt;ffffffffa0574cc8&gt;] class_setup+0x208/0x870 [obdclass] [&lt;ffffffffa057c59c&gt;] class_process_config+0x113c/0x27c0 [obdclass] [&lt;ffffffff8117008c&gt;] ? __kmalloc+0x20c/0x220 [&lt;ffffffff8117008c&gt;] ? __kmalloc+0x20c/0x220 [&lt;ffffffffa0556a65&gt;] class_handle_ioctl+0xbc5/0x21b0 [obdclass] [&lt;ffffffffa053e2ab&gt;] obd_class_ioctl+0x4b/0x190 [obdclass] [&lt;ffffffff8119dfc2&gt;] vfs_ioctl+0x22/0xa0 [&lt;ffffffff8119e164&gt;] do_vfs_ioctl+0x84/0x580 [&lt;ffffffff8119e6e1&gt;] sys_ioctl+0x81/0xa0 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>and if required, I can provide a test to be added in llog_test to create the catalog full situation.</p> <p>this is obvious bug, log_destroy() shouldn't be called in error path in this function because it starts new transaction while in context of another one.</p> <p>Since this llog cannot be deleted inside current transaction, I propose to mark it as 'invalid' or 'orphaned' and delete outside this call, probably in llog processing. It can be also deleted by lfsck in case of server reboot also by checking such flag.</p> <p>This can be fixed now with simple fix after commit made by Wang Di for llog changes related to DNE. We have to use just llog_trans_destroy() there.</p> <p>Actually the patch on <a href="https://jira.whamcloud.com/browse/LU-6846" title="dt_record_write()) ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed: " class="issue-link" data-issue-key="LU-6846"><del>LU-6846</del></a> will fix this issue.<a href="http://review.whamcloud.com/16333" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16333</a></p> <p>The idea is that we do not need take care of the created llog object, because if llog_cat_new_log() fails, and the log handle has been added to phd_entry, so llog_cat_close() will take care of the created object anyway.</p> <p>Di, llog_cat_close() is called at mount. meaning llog_cat_new_log() leaves an orphan?</p> <p>Alex Zhuravlev (alexey.zhuravlev@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16427" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16427</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6634" title="(osd_handler.c:901:osd_trans_start()) ASSERTION( get_current()-&gt;journal_info == ((void *)0) ) failed: when reaching Catalog full condition" class="issue-link" data-issue-key="LU-6634"><del>LU-6634</del></a> llog: destroy plain llog if init fails<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7edf0a47546b03ac886500d1b78f1cb01084850e</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Di, llog_cat_close() is called at mount. meaning llog_cat_new_log() leaves an orphan? </pre> </div></div> <p>hmm, not sure I follow, you mean called at umount? why can not this code take care of this?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> list_for_each_entry_safe(loghandle, n, &amp;cathandle-&gt;u.chd.chd_head, u.phd.phd_entry) { struct llog_log_hdr *llh = loghandle-&gt;lgh_hdr; int index; /* unlink open-not-created llogs */ list_del_init(&amp;loghandle-&gt;u.phd.phd_entry); llh = loghandle-&gt;lgh_hdr; if (loghandle-&gt;lgh_obj != NULL &amp;&amp; llh != NULL &amp;&amp; (llh-&gt;llh_flags &amp; LLOG_F_ZAP_WHEN_EMPTY) &amp;&amp; (llh-&gt;llh_count == 1)) { rc = llog_destroy(env, loghandle); </pre> </div></div> <p>because this code is called at umount (when we close the llog). and of course, this is very different transaction.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>because this code is called at umount (when we close the llog). and of course, this is very different transaction. </pre> </div></div> <p>Well, it is not an orphan, and will be destroyed during umount. TBH, I do not think it is a big deal. The reason I am a bit reluctant is that if you do destroy here, you have to reserve the credit for this destroy here? Btw: I did not see it declare the destroy in llog_cat_decalre_add_rec()? </p> <p>there is no need to declare that as it's a reverse operation and it doesn't consume additional credits. it's an orphan because it isn't referenced by some on-disk structure. so if we crash before umount this object is leaked. I agree this isn't a very big deal, but if this can be fixed easily, why not?</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16427/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16427/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6634" title="(osd_handler.c:901:osd_trans_start()) ASSERTION( get_current()-&gt;journal_info == ((void *)0) ) failed: when reaching Catalog full condition" class="issue-link" data-issue-key="LU-6634"><del>LU-6634</del></a> llog: destroy plain llog if init fails<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9126d8159d5d5b61a600e7427d0c173084a710e6</p> <p>Landed for 2.8</p> Related LU-6556 LU-7138 LU-6954 Development Rank 1|hzxe3b: Rank (Obsolete) 9223372036854775807 Severity