https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2279 Lustre <p>Running replay-single, I hit this LBUG:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[50576.766529] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files == 01:10:48 (1352095848) [50576.964168] Turning device loop1 (0x700001) read-only [50576.985608] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [50576.989899] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 [50577.290548] Turning device loop0 (0x700000) read-only [50577.316807] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [50577.323211] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [50577.541027] Lustre: lustre-OST0000-osc-MDT0000: slow creates, last=8104, next=8104, reserved=0, syn_changes=0, syn_rpc_in_progress=0, status=-19 [50577.541472] LustreError: 10305:0:(osp_precreate.c:694:osp_precreate_reserve()) LBUG [50577.541985] Pid: 10305, comm: mdt00_000 [50577.542331] [50577.542332] Call Trace: [50577.542866] [&lt;ffffffffa0455915&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [50577.543256] [&lt;ffffffffa0455f27&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [50577.543618] [&lt;ffffffffa0e4473c&gt;] osp_precreate_reserve+0x50c/0x600 [osp] [50577.544000] [&lt;ffffffffa0e3d7a0&gt;] ? osp_object_alloc+0x120/0x290 [osp] [50577.544368] [&lt;ffffffffa0e43533&gt;] osp_declare_object_create+0x163/0x540 [osp] [50577.544785] [&lt;ffffffffa07384bd&gt;] lod_qos_declare_object_on+0xed/0x4c0 [lod] [50577.545173] [&lt;ffffffffa073b65b&gt;] lod_qos_prep_create+0x92b/0x1848 [lod] [50577.545549] [&lt;ffffffffa073686b&gt;] lod_declare_striped_object+0x14b/0x920 [lod] [50577.546558] [&lt;ffffffffa0737348&gt;] lod_declare_object_create+0x308/0x4f0 [lod] [50577.546905] [&lt;ffffffffa0cf1c4f&gt;] mdd_declare_object_create_internal+0xaf/0x1d0 [mdd] [50577.547452] [&lt;ffffffffa0d02aca&gt;] mdd_create+0x39a/0x1550 [mdd] [50577.547770] [&lt;ffffffffa0de9d28&gt;] mdt_reint_open+0x1078/0x1870 [mdt] [50577.548094] [&lt;ffffffffa0d08b0e&gt;] ? md_ucred+0x1e/0x60 [mdd] [50577.548403] [&lt;ffffffffa0db5655&gt;] ? mdt_ucred+0x15/0x20 [mdt] [50577.548718] [&lt;ffffffffa0dd4c81&gt;] mdt_reint_rec+0x41/0xe0 [mdt] [50577.549059] [&lt;ffffffffa0dce413&gt;] mdt_reint_internal+0x4e3/0x7e0 [mdt] [50577.549395] [&lt;ffffffffa0dce9dd&gt;] mdt_intent_reint+0x1ed/0x500 [mdt] [50577.549715] [&lt;ffffffffa0dca3ae&gt;] mdt_intent_policy+0x38e/0x770 [mdt] [50577.550077] [&lt;ffffffffa0783dda&gt;] ldlm_lock_enqueue+0x2ea/0x890 [ptlrpc] [50577.550424] [&lt;ffffffffa07ab187&gt;] ldlm_handle_enqueue0+0x4e7/0x1010 [ptlrpc] [50577.550761] [&lt;ffffffffa0dca8e6&gt;] mdt_enqueue+0x46/0x130 [mdt] [50577.551076] [&lt;ffffffffa0dbe1f2&gt;] mdt_handle_common+0x932/0x1740 [mdt] [50577.551425] [&lt;ffffffffa0dbf0d5&gt;] mdt_regular_handle+0x15/0x20 [mdt] [50577.551797] [&lt;ffffffffa07d9573&gt;] ptlrpc_server_handle_request+0x463/0xe70 [ptlrpc] [50577.552413] [&lt;ffffffffa045666e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [50577.552697] [&lt;ffffffffa07d2241&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] [50577.552950] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [50577.553303] [&lt;ffffffffa07dc10a&gt;] ptlrpc_main+0xb9a/0x1960 [ptlrpc] [50577.553709] [&lt;ffffffffa07db570&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [50577.553927] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [50577.554222] [&lt;ffffffffa07db570&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [50577.554605] [&lt;ffffffffa07db570&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [50577.554960] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 [50577.555289] [50577.560016] Kernel panic - not syncing: LBUG </pre> </div></div> <p>crashdump and modules are at /exports/crashdumps/192.168.10.210-2012-11-05-01\:10\:50</p> LU-2279

replay-single test 88 LustreError: 10305:0:(osp_precreate.c:694:osp_precreate_reserve()) LBUG

Bug Blocker Resolved Fixed Alex Zhuravlev Oleg Drokin MB sequoia Mon, 5 Nov 2012 15:24:24 +0000 Fri, 19 Apr 2013 20:48:02 +0000 Tue, 8 Jan 2013 23:00:46 +0000 Lustre 2.4.0 Lustre 2.4.0 0 8 <p>Hit this on our test MDS after updating to our <tt>2.3.56-1chaos</tt> tag.</p> <p>Bump. Hit again on reboot.</p> <p>Prakash, are you getting this on replay-single/88 or on a regular (not driven by test-framework) umount ?</p> <p>Not driven by test-framework. My cases were on our Grove (development) filesystem.</p> <p>Alex, we also hit this in DNE tests, with 4MDS/8MDT. It seems busy loop check should not include NODEV cases (i.e. ost is not being setup in time). Do you think this patch reasonable?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/lustre/osp/osp_precreate.c b/lustre/osp/osp_precreate.c index 62b1ef5..48ee758 100644 --- a/lustre/osp/osp_precreate.c +++ b/lustre/osp/osp_precreate.c @@ -1066,19 +1066,17 @@ int osp_precreate_reserve(const struct lu_env *env, struct osp_device *d) if (unlikely(rc == -ENODEV)) { if (cfs_time_aftereq(cfs_time_current(), expire)) break; - } + } else if (unlikely(count++ == 1000)) { #if LUSTRE_VERSION_CODE &lt; OBD_OCD_VERSION(2, 3, 90, 0) - /* - * to address Andreas's concern on possible busy-loop - * between this thread and osp_precreate_send() - */ - if (unlikely(count++ == 1000)) { + /* + * to address Andreas's concern on possible busy-loop + * between this thread and osp_precreate_send() + */ osp_precreate_timeout_condition(d); LBUG(); - } #endif - + } /* * increase number of precreations */ </pre> </div></div> <p>well, the problem is not in ENODEV itself, but how often we're awaken. if we do not wait and no alive OST is found, then file creation fails.<br/> the question is why (or who) wakes this thread up. can you investigate please ?</p> <p>Hmm, if precreate thread should not be wake up in this case, then probably osp_precreate_ready_condition should be fixed ? (skip -ENODEV for second if?)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static int osp_precreate_ready_condition(const struct lu_env *env, struct osp_device *d) { /* ready if got enough precreated objects */ /* we need to wait for others (opd_pre_reserved) and our object (+1) */ if (d-&gt;opd_pre_reserved + 1 &lt; osp_objs_precreated(env, d)) return 1; /* ready if OST reported no space and no destoys in progress */ if (d-&gt;opd_syn_changes + d-&gt;opd_syn_rpc_in_progress == 0 &amp;&amp; d-&gt;opd_pre_status != 0) return 1; return 0; } </pre> </div></div> <p>It seems it will always return 1 on second if, because syn_changes and rpc_in_progress are zero at this moment, and pre_status == -ENODEV.</p> <p>but we do not want to be blocked forever. this is why osp_precreate_reserve() exits when specified period expires. again the question why is it woken up so often?<br/> given there is no connection, why status has changed that many times?</p> <p>I am confused here, or I miss something probably. According to the log we saw here</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[50577.541027] Lustre: lustre-OST0000-osc-MDT0000: slow creates, last=8104, next=8104, reserved=0, syn_changes=0, syn_rpc_in_progress=0, status=-19 </pre> </div></div> <p>sync_changes and syn_rpc_in_progress are zero as expected, and status is -19 because ost is not setup in time, so the loop in osp_precreate_reserve will<br/> not sleep at all (why you say it was waken up so often?), i.e. it will keep running until count reach 1000 and hit LBUG, unless it reaches expire time first, which might be unlikely given that expire time is normally 100 seconds.</p> <p>Another thing I do not understand is that why this OST is still being chosen to do procreate if it is not ready yet, probably another bug?</p> <p>Please correct me if I misunderstand sth.</p> <p>yes, sorry for confusion.. i totally forgot my own code. there was a problem long ago when OSP ss reconnecting too frequently leading to a storm of wakeups.</p> <p>as for OST being selected - it's valid as another OSTs might be in the same state (or just empty, for example).<br/> there is also a race between disconnect and selection procedure.</p> <p>to understand whether this was bad choice or not it's important to know other OST's states.</p> <p><a href="http://review.whamcloud.com/4847" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4847</a></p> <p>the patch is landed</p> Development Rank 1|hzvbpr: Rank (Obsolete) 5462 Severity