https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7975 Lustre <p>A site has encountered multiple crashes with same signature/stack+msgs following :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 89879:0:(osp_precreate.c:1222:osp_object_truncate()) can't punch object: -11 Lustre: composit-OST0009-osc-MDT0000: Connection to composit-OST0009 (at 10.0.14.31@o2ib) was lost; in progress operations using this service will wait for recovery to complete LustreError: 89879:0:(lod_object.c:700:lod_ah_init()) ASSERTION( lc-&gt;ldo_stripenr == 0 ) failed: LustreError: 89879:0:(lod_object.c:700:lod_ah_init()) LBUG Pid: 89879, comm: mdt01_006 Call Trace: [&lt;ffffffffa057e895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa057ee97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa266c0af&gt;] lod_ah_init+0x58f/0x5d0 [lod] [&lt;ffffffffa26c7ad3&gt;] mdd_object_make_hint+0x83/0xa0 [mdd] [&lt;ffffffffa26d4502&gt;] mdd_create_data+0x332/0x7d0 [mdd] [&lt;ffffffffa25a93f0&gt;] mdt_finish_open+0x1350/0x19a0 [mdt] [&lt;ffffffffa257e5f4&gt;] ? mdt_object_lock+0x14/0x20 [mdt] [&lt;ffffffffa25a9fbd&gt;] mdt_open_by_fid_lock+0x57d/0x910 [mdt] [&lt;ffffffffa25aabac&gt;] mdt_reint_open+0x56c/0x21a0 [mdt] [&lt;ffffffffa059b14c&gt;] ? upcall_cache_get_entry+0x29c/0x890 [libcfs] [&lt;ffffffffa0983930&gt;] ? lu_ucred+0x20/0x30 [obdclass] [&lt;ffffffffa2572945&gt;] ? mdt_ucred+0x15/0x20 [mdt] [&lt;ffffffffa258f8ec&gt;] ? mdt_root_squash+0x2c/0x410 [mdt] [&lt;ffffffffa123bad6&gt;] ? __req_capsule_get+0x166/0x710 [ptlrpc] [&lt;ffffffffa2593ab1&gt;] mdt_reint_rec+0x41/0xe0 [mdt] [&lt;ffffffffa2578f83&gt;] mdt_reint_internal+0x4c3/0x780 [mdt] [&lt;ffffffffa257950e&gt;] mdt_intent_reint+0x1ee/0x520 [mdt] [&lt;ffffffffa2576cee&gt;] mdt_intent_policy+0x3ae/0x770 [mdt] [&lt;ffffffffa11ca2f5&gt;] ldlm_lock_enqueue+0x135/0x980 [ptlrpc] [&lt;ffffffffa11f43fb&gt;] ldlm_handle_enqueue0+0x51b/0x10c0 [ptlrpc] [&lt;ffffffffa25771b6&gt;] mdt_enqueue+0x46/0xe0 [mdt] [&lt;ffffffffa257c84a&gt;] mdt_handle_common+0x52a/0x1470 [mdt] [&lt;ffffffffa25b98f5&gt;] mds_regular_handle+0x15/0x20 [mdt] [&lt;ffffffffa12238d5&gt;] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] [&lt;ffffffffa05904fa&gt;] ? lc_watchdog_touch+0x7a/0x190 [libcfs] [&lt;ffffffffa121c289&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] [&lt;ffffffff81057849&gt;] ? __wake_up_common+0x59/0x90 [&lt;ffffffffa122605d&gt;] ptlrpc_main+0xaed/0x1780 [ptlrpc] [&lt;ffffffffa1225570&gt;] ? ptlrpc_main+0x0/0x1780 [ptlrpc] [&lt;ffffffff8109e78e&gt;] kthread+0x9e/0xc0 [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 [&lt;ffffffff8109e6f0&gt;] ? kthread+0x0/0xc0 [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>According to existing tickets, I have found that this kind of problem has already (partially?) been addressed in LU&#45;4260, LU&#45;4791 and LU&#45;5346 tickets. <br/> And since both fixes for LU&#45;4260 and LU&#45;4791 are already integrated, this means that we encounter a new situation/problem during OST objects pre-creation, likely to be caused by some specific file meta-data pattern (I have identified as "deferred layout" feature usage with open(, ...|O_LOV_DELAY_CREATE|...,) along with a non-0 truncate() to trigger objects preallocation), leading to trigger a similar case than described in LU&#45;5346 upon error return path that is still not fixed.</p> <p>BTW, I have also determined that these MDT assert always occurs just following an OSS crash, hence the &#45;EAGAIN/EWOULDBLOCK error in "(osp_precreate.c:1222:osp_object_truncate()) can't punch object: -11" msg just preceding the assert !</p> LU-7975

"(lod_object.c:700:lod_ah_init()) ASSERTION( lc->ldo_stripenr == 0 )" LBUG/Assert on MDS

Bug Minor Resolved Fixed Bruno Faccini Bruno Faccini cea Fri, 1 Apr 2016 15:35:17 +0000 Thu, 14 Jun 2018 21:41:17 +0000 Tue, 31 May 2016 12:50:05 +0000 Lustre 2.9.0 0 10 <p>the good thing is that we've got a new infrastructure with DNE2 allowing for striping procedures (declaration and creation) to be per-thandle and not put intermediate state into an object. another improvement we can make with this infrastructure is to release OSP objects after creation, don't leave them in cache consuming memory.</p> <p>After more crash-dump and Lustre debug logs analysis, I was able to find a reproducer for the situation unveiled at customer site and first causing MDS CERROR/msg "(osp_precreate.c:1222:osp_object_truncate()) can't punch object: &#45;11" and a few later "(lod_object.c:700:lod_ah_init()) ASSERTION( lc-&gt;ldo_stripenr == 0 ) failed" Assertion/LBUG() :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># !!! with all OSTs down !!! mkdir /mnt/lustre/foo_dir/ lfs setstripe -c 2 /mnt/lustre/foo_dir/ # open file with O_LOV_DELAY_CREATE, then non-0 truncate multiop /mnt/lustre/foo_dir/foo5 oO_RDWR:O_CREAT:O_LOV_DELAY_CREATE:T1050000c # normal open cat /etc/hosts &gt; /mnt/lustre/foo_dir/foo5 ---&gt;&gt;&gt; "-bash: /mnt/lustre/foo_dir/foo5: Resource temporarily unavailable" and "(osp_precreate.c:1222:osp_object_truncate()) can't punch object: -11" # normal open again cat /etc/hosts &gt; /mnt/lustre/foo_dir/foo5 ---&gt;&gt;&gt; "LustreError: 11915:0:(lod_object.c:700:lod_ah_init()) ASSERTION( lc-&gt;ldo_stripenr == 0 ) failed:" </pre> </div></div> <p>Then, diging more in both MDS full Lustre debug log along into the source code, I found the reason of the -EAGAIN return from osp_object_create() call, which is not sent over the wire but in turn is coded as -EWOULDBLOCK in ptlrpc_import_delay_req() !! :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>/** * Based on the current state of the import, determine if the request * can be sent, is an error, or should be delayed. * * Returns true if this request should be delayed. If false, and * *status is set, then the request can not be sent and *status is the * error code. If false and status is 0, then request can be sent. * * The imp-&gt;imp_lock must be held. */ static int ptlrpc_import_delay_req(struct obd_import *imp, struct ptlrpc_request *req, int *status) { int delay = 0; ENTRY; .............. } else if (req-&gt;rq_send_state != imp-&gt;imp_state) { /* invalidate in progress - any requests should be drop */ if (cfs_atomic_read(&amp;imp-&gt;imp_inval_count) != 0) { DEBUG_REQ(D_ERROR, req, "invalidate in flight"); *status = -EIO; } else if (imp-&gt;imp_dlm_fake || req-&gt;rq_no_delay) { *status = -EWOULDBLOCK; &lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt; ......................... </pre> </div></div> <p>when</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>/* * */ int osp_object_truncate(const struct lu_env *env, struct dt_object *dt, __u64 size) { ............... /* * XXX: decide how do we do here with resend * if we don't resend, then client may see wrong file size * if we do resend, then MDS thread can get stuck for quite long */ req-&gt;rq_no_resend = req-&gt;rq_no_delay = 1; ................ </pre> </div></div> <p>So this raises more implications and I think resend (or any retry mechanism) may not be an option at the moment if we don't want data corruption failure (missing non-0 truncate() effect/result) when this particular scenario occurs. Because even if striping allocated things/stuff cleanup upon error return path must be fixed in MDS code, the error should not be returned to the user otherwise further open() will fail and non-0 truncate() effect/result will be lost.</p> <p>as for truncate, there is a ticket to move this functionality to the client side. could be a good time, probably..</p> <p>Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/19301" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19301</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7975" title="&quot;(lod_object.c:700:lod_ah_init()) ASSERTION( lc-&gt;ldo_stripenr == 0 )&quot; LBUG/Assert on MDS" class="issue-link" data-issue-key="LU-7975"><del>LU-7975</del></a> lod: fix delayed stripe error path &amp; MDS resend<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 43a47da70a47d027f9ac199ed4ff6fdc4fe91614</p> <p>Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/19302" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19302</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7975" title="&quot;(lod_object.c:700:lod_ah_init()) ASSERTION( lc-&gt;ldo_stripenr == 0 )&quot; LBUG/Assert on MDS" class="issue-link" data-issue-key="LU-7975"><del>LU-7975</del></a> lod: fix delayed stripe error path &amp; Client resend<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bb8a376656f6b3626a47c32cbc1855045a47d929</p> <p>Patch at <a href="http://review.whamcloud.com/19301" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19301</a> fixes cleanup in delayed stripe error path and also implements resend mechanism from MDS side. It may keep a MDS thread busy for some time doing so.</p> <p>Patch at <a href="http://review.whamcloud.com/19302" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19302</a> is an other way to fix, also doing delayed stripe error path necessary cleanup, but offloading resend mechanism to Client side, which may be less intrusive.</p> <p><a href="http://review.whamcloud.com/19301" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19301</a> has been abandoned in favor of <a href="http://review.whamcloud.com/19302" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19302</a>, according to reviewers comments and choice between both solutions.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19302/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19302/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7975" title="&quot;(lod_object.c:700:lod_ah_init()) ASSERTION( lc-&gt;ldo_stripenr == 0 )&quot; LBUG/Assert on MDS" class="issue-link" data-issue-key="LU-7975"><del>LU-7975</del></a> lod: fix delayed stripe error path &amp; Client resend<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 047dfe489966c8816cbead1a3abbbb1564fdb7db</p> <p>Landed for 2.9</p> Related LU-4971 Development Rank 1|hzy6n3: Rank (Obsolete) 9223372036854775807 Severity