https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14277 Lustre <p>multiple MDT threads blocked by one declare_create call due OST fail.<br/> rwsem_down_write_failed<br/> call_rwsem_down_write_failed<br/> lod_qos_prep_create<br/> lod_declare_striped_object<br/> lod_declare_object_create<br/> mdd_declare_object_create_internal<br/> mdd_declare_create<br/> mdd_create<br/> mdo_create<br/> mdt_reint_open<br/> mdt_reint_rec<br/> mdt_reint_internal<br/> mdt_intent_reint<br/> mdt_intent_policy<br/> ldlm_lock_enqueue<br/> ldlm_handle_enqueue0<br/> tgt_enqueue<br/> tgt_request_handle<br/> ptlrpc_main<br/> kthread<br/> vs<br/> osp_precreate_reserve<br/> osp_declare_object_create<br/> lod_qos_declare_object_on<br/> lod_alloc_rr.clone.2<br/> lod_qos_prep_create<br/> lod_declare_striped_object<br/> lod_declare_object_create<br/> mdd_declare_object_create_internal<br/> mdd_declare_create<br/> mdd_create<br/> mdo_create<br/> mdt_reint_open<br/> mdt_reint_rec<br/> mdt_reint_internal<br/> mdt_intent_reint<br/> mdt_intent_policy<br/> ldlm_lock_enqueue<br/> ldlm_handle_enqueue0<br/> tgt_enqueue<br/> tgt_request_handle<br/> ptlrpc_main<br/> kthread<br/> kernel_thread</p> LU-14277

any create blocked due any OST fail

Bug Blocker Resolved Fixed Alexey Lyashkov Alexey Lyashkov Wed, 23 Dec 2020 13:51:29 +0000 Mon, 12 Jun 2023 23:52:32 +0000 Wed, 26 Jan 2022 14:37:52 +0000 Lustre 2.15.0 0 8 <p>I think this is a dup of <a href="https://jira.whamcloud.com/browse/LU-13073" title="Multiple MDS deadlocks (in lod_qos_prep_create) after OSS crash" class="issue-link" data-issue-key="LU-13073"><del>LU-13073</del></a></p> <p>It probably. But my fix much much simple - just revert some code landed with LOV -&gt; LOD transmission.<br/> Let's looks to it.</p> <p>one more bug introduced by last changes in replace LQ_DIRTY with bitfield.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>c1d0a355a6a (Lai Siyao 2019-08-05 02:08:02 +0800 327) up_write(&amp;ltd-&gt;ltd_qos.lq_rw_sem); c7f2e70a27e (Alex Zhuravlev 2012-09-20 15:56:31 +0400 328) c7f2e70a27e (Alex Zhuravlev 2012-09-20 15:56:31 +0400 329) if (placed != real_count) { c7f2e70a27e (Alex Zhuravlev 2012-09-20 15:56:31 +0400 330) /* This should never happen */ c1d0a355a6a (Lai Siyao 2019-08-05 02:08:02 +0800 331) LCONSOLE_ERROR_MSG(0x14e, "Failed to place all tgts in the " c7f2e70a27e (Alex Zhuravlev 2012-09-20 15:56:31 +0400 332) "round-robin list (%d of %d).\n", c7f2e70a27e (Alex Zhuravlev 2012-09-20 15:56:31 +0400 333) placed, real_count); c7f2e70a27e (Alex Zhuravlev 2012-09-20 15:56:31 +0400 334) for (i = 0; i &lt; lqr-&gt;lqr_pool.op_count; i++) { c1d0a355a6a (Lai Siyao 2019-08-05 02:08:02 +0800 335) LCONSOLE(D_WARNING, "rr #%d tgt idx=%d\n", i, c7f2e70a27e (Alex Zhuravlev 2012-09-20 15:56:31 +0400 336) lqr-&gt;lqr_pool.op_array[i]); c7f2e70a27e (Alex Zhuravlev 2012-09-20 15:56:31 +0400 337) } c7f2e70a27e (Alex Zhuravlev 2012-09-20 15:56:31 +0400 338) lqr-&gt;lqr_dirty = 1; c7f2e70a27e (Alex Zhuravlev 2012-09-20 15:56:31 +0400 339) RETURN(-EAGAIN); </pre> </div></div> <p>lqr_dirty was without lock held.</p> <blockquote> <p>But my fix much much simple - just revert some code landed with LOV -&gt; LOD transmission.</p></blockquote> <p>Why not push that patch under <a href="https://jira.whamcloud.com/browse/LU-13073" title="Multiple MDS deadlocks (in lod_qos_prep_create) after OSS crash" class="issue-link" data-issue-key="LU-13073"><del>LU-13073</del></a>?</p> <blockquote> <p>lqr_dirty was without lock held.</p></blockquote> <p>That <tt>lqr_dirty = 1</tt> is in a "should never happen" code path with a very clear error message that is not being seen here, so it doesn't seem to be related to this problem.</p> <blockquote> <p>one more bug introduced by last changes in replace LQ_DIRTY with bitfield.</p></blockquote> <p>I don't see any patch like that... There no reference to "<tt>LQ_DIRTY</tt>" in any of the patches in Git. The very first patch introducing <tt>lqr_dirty</tt> (replacing <tt>lq_dirty_rr</tt>) is using a bitfield, and it has been a bitfield since that time, even though the code was moved around to different files a few times:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 665e36b780faa2144cecccd29a0d8a8196a76903 Author: Nathan Rutman &lt;nathan@clusterfs.com&gt; AuthorDate: Mon Sep 15 23:25:34 2008 +0000 b=14836 i=nathan i=adilger OST pools on HEAD, comprehensive patch including 17054:19007; 16935:18918,19012,19089,19128; 16978:18872 : : +/* Round-robin allocator data */ +struct lov_qos_rr { + __u32 lqr_start_idx; /* start index of new inode */ + __u32 lqr_offset_idx; /* aliasing for start_idx */ + int lqr_start_count; /* reseed counter */ + struct ost_pool lqr_pool; /* round-robin optimized list */ + unsigned long lqr_dirty:1; /* recalc round-robin list */ +}; + +/* Stripe placement optimization */ struct lov_qos { struct list_head lq_oss_list; /* list of OSSs that targets use */ struct rw_semaphore lq_rw_sem; __u32 lq_active_oss_count; - __u32 *lq_rr_array; /* round-robin optimized list */ - unsigned int lq_rr_size; /* rr array size */ unsigned int lq_prio_free; /* priority for free space */ + struct lov_qos_rr lq_rr; /* round robin qos data */ unsigned long lq_dirty:1, /* recalc qos data */ - lq_dirty_rr:1, /* recalc round-robin list */ lq_same_space:1,/* the ost's all have approx. the same space avail */ lq_reset:1; /* zero current penalties */ </pre> </div></div> <p>The patch that introduced <tt>lq_dirty_rr</tt> was also using a bitfield:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 113303973ec9f8484eb2355a1a6ef3c4c7fd6a56 Author: Nathan Rutman &lt;nathan@clusterfs.com&gt; AuthorDate: Sat Feb 10 06:33:41 2007 +0000 land b1_5 onto HEAD : : +struct lov_qos { + struct list_head lq_oss_list; /* list of OSSs that targets use */ + struct rw_semaphore lq_rw_sem; + __u32 lq_active_oss_count; + __u32 *lq_rr_array; /* round-robin optimized list */ + unsigned int lq_rr_size; /* rr array size */ + unsigned int lq_prio_free; /* priority for free space */ + unsigned int lq_dirty:1, /* recalc qos data */ + lq_dirty_rr:1, /* recalc round-robin list */ + lq_same_space:1,/* the ost's all have approx. + the same space avail */ + lq_reset:1; /* zero current penalties */ +}; </pre> </div></div> <p>I'm happy to see a simpler fix and look forward to seeing your patch.</p> <p>Andreas, lets open a lustre 2.0..2.4 code. Before an LOD was introduced.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@lustre lov]# grep _DIRTY * lov_obd.c: cfs_set_bit(LQ_DIRTY, &amp;lov-&gt;lov_qos.lq_flags); lov_qos.c: cfs_set_bit(LQ_DIRTY, &amp;lov-&gt;lov_qos.lq_flags); lov_qos.c: cfs_set_bit(LQ_DIRTY, &amp;lov-&gt;lov_qos.lq_flags); lov_qos.c: if (!cfs_test_bit(LQ_DIRTY, &amp;lov-&gt;lov_qos.lq_flags)) lov_qos.c: cfs_clear_bit(LQ_DIRTY, &amp;lov-&gt;lov_qos.lq_flags); lov_qos.c: if (!cfs_test_bit(LQ_DIRTY, &amp;lov-&gt;lov_qos.lq_flags) &amp;&amp; lov_qos.c: if (cfs_test_bit(LQ_DIRTY, &amp;lov-&gt;lov_qos.lq_flags)) lov_qos.c: cfs_set_bit(LQ_DIRTY, &amp;lov-&gt;lov_qos.lq_flags); lproc_lov.c: cfs_set_bit(LQ_DIRTY, &amp;lov-&gt;lov_qos.lq_flags); lproc_lov.c: cfs_set_bit(LQ_DIRTY, &amp;lov-&gt;lov_qos.lq_flags); </pre> </div></div> <p>But i talk about LQ_SF_PROGRESS bit.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>void qos_statfs_update(struct obd_device *obd, __u64 max_age, int wait) { struct lov_obd *lov = &amp;obd-&gt;u.lov; struct obd_info *oinfo; int rc = 0; struct ptlrpc_request_set *set = NULL; ENTRY; if (cfs_time_beforeq_64(max_age, obd-&gt;obd_osfs_age)) /* statfs data are quite recent, don't need to refresh it */ RETURN_EXIT; /* Check if statfs already in progress */ if (cfs_test_and_set_bit(LQ_SF_PROGRESS, &amp;lov-&gt;lov_qos.lq_flags)) GOTO(out, rc = 0); </pre> </div></div> <p>LQ_SF_PROGRESS was gone in LOV -&gt; LOD transmission, but LQ_DIRTY still exist in 2.7..2.12.<br/> And gone with QOS move from lod to the lu_qos.</p> <p>for 2.12 code this code is safe.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> --- a/lustre/lod/lod_internal.h +++ b/lustre/lod/lod_internal.h @@ -83,7 +83,7 @@ <span class="code-keyword">enum</span> lq_flag { LQ_SAME_SPACE = 1, /* the ost's all have approx. the same space avail */ LQ_RESET = 2, <span class="code-comment">/* zero current penalties */</span> - + LQ_SF_PROGRESS = 3 <span class="code-comment">/* statfs op in progress */</span> }; struct lod_qos { diff --git a/lustre/lod/lod_qos.c b/lustre/lod/lod_qos.c index 1c377422b1..be0c10f0e8 100644 --- a/lustre/lod/lod_qos.c +++ b/lustre/lod/lod_qos.c @@ -291,11 +291,11 @@ void lod_qos_statfs_update(<span class="code-keyword">const</span> struct lu_env *env, struct lod_device *lod) <span class="code-comment">/* statfs data are quite recent, don't need to refresh it */</span> RETURN_EXIT; - down_write(&amp;lod-&gt;lod_qos.lq_rw_sem); - - <span class="code-keyword">if</span> (obd-&gt;obd_osfs_age &gt; max_age) - <span class="code-keyword">goto</span> out; + <span class="code-comment">/* Check <span class="code-keyword">if</span> statfs already in progress */</span> + <span class="code-keyword">if</span> (test_and_set_bit(LQ_SF_PROGRESS, &amp;lod-&gt;lod_qos.lq_flags)) + RETURN_EXIT; + down_read(&amp;osts-&gt;op_rw_sem); <span class="code-keyword">for</span> (i = 0; i &lt; osts-&gt;op_count; i++) { idx = osts-&gt;op_array[i]; avail = OST_TGT(lod,idx)-&gt;ltd_statfs.os_bavail; @@ -306,10 +306,12 @@ void lod_qos_statfs_update(<span class="code-keyword">const</span> struct lu_env *env, struct lod_device *lod) <span class="code-comment">/* recalculate weigths */</span> set_bit(LQ_DIRTY, &amp;lod-&gt;lod_qos.lq_flags); } + up_read(&amp;osts-&gt;op_rw_sem); + obd-&gt;obd_osfs_age = ktime_get_seconds(); -out: - up_write(&amp;lod-&gt;lod_qos.lq_rw_sem); + clear_bit(LQ_SF_PROGRESS, &amp;lod-&gt;lod_qos.lq_flags); + EXIT; } </pre> </div></div> <p>read semaphore is good protection from the global pool modification and set dirty bit is good atomic operation.<br/> But for current master i stick with porting. Because i want a revert so much code to have working it fine.</p> <p>Hm.. it looks this patch just don't accepted by WC in past :-/<br/> <a href="https://jira.whamcloud.com/browse/LU-7853" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-7853</a></p> <p>there was an attempt to convert those bitfields to atomic bits : <a href="https://jira.whamcloud.com/browse/LU-7853" title="Fixes bitfield in lod qos code" class="issue-link" data-issue-key="LU-7853"><del>LU-7853</del></a> , needs to be updated after lod_qos code reorganization.</p> <blockquote> <p>Hm.. it looks this patch just don't accepted by WC in past :-/</p></blockquote> <p>The original version of <a href="https://jira.whamcloud.com/browse/LU-7853" title="Fixes bitfield in lod qos code" class="issue-link" data-issue-key="LU-7853"><del>LU-7853</del></a> didn't give any reason why the patch was needed, so I asked for an explanation and eventually got one. Then the patch was failing testing repeatedly until it was fixed. I've reviewed that patch many times, and approved it. The last version finished testing the day before another batch of patches were landed by Oleg, but it didn't get refreshed since that time. </p> <p>I don't think that equals "don't accepted by WC". Everyone has to follow the same process for landing patches. I have lots of patches that I would like landed, but I know the work is mine to update them and to get them past testing before they can land. </p> <p>I think the rule is very simple: patch accepted - you're lucky, grab a cake; not yet accepted - press rebase.</p> <p>Andreas, this patch is needs because of large number points where unprotected set used.<br/> If any parallel set to same 'long' will used this will be lost one of modifications due read-modify-set operation.<br/> set_bit is prevent such bugs.<br/> Anyway, Thanks Zam to the explanation why i seen this patch on any own tree for any 2.x based and it looks time to rebase and fix last issues with him. Like lqr_dirty and lq_dirty is same semantic but it one of them don't set sometimes (like with add target to pool).</p> <p>Alex, sorry about reverting notice.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=shadow" class="user-hover" rel="shadow">shadow</a>I'm fine, I'm just trying to say that the list of the patches (ready to land!) can be very long and sometimes it takes months to get through - because of priorities, luck, etc. I do have patches hanging around with +2 for months, but basically master can't accept too many in short time, it's just a way to keep it more-less stable. the good thing is that rebase button simplifies efforts to maintain patch uptodate (not guaranteed, of course). and a good test demonstrating the issue helps, obviously.</p> <p>Ok. i will pickup an <a href="https://jira.whamcloud.com/browse/LU-7853" title="Fixes bitfield in lod qos code" class="issue-link" data-issue-key="LU-7853"><del>LU-7853</del></a> rebase it, and fix an problem with qos_rr lqr dirty. It looks it need to be covered to bit flag also.</p> <p><a href="https://review.whamcloud.com/#/c/41497/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/41497/</a></p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/41497/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41497/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14277" title="any create blocked due any OST fail" class="issue-link" data-issue-key="LU-14277"><del>LU-14277</del></a> lod: statfs should not block a create<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6cdb08a0c598b20a4b0ce27f7fe4763b0ada3118</p> <p>Landed for 2.15</p> Duplicate LU-13073 Related LU-7853 LU-15393 Development Rank 1|i01i5r: Rank (Obsolete) 9223372036854775807 Severity