https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2877 Lustre <p>Happens pretty frequently in maloo since April 2012.<br/> Sample failures:<br/> <a href="https://maloo.whamcloud.com/test_sets/33b06c22-8065-11e2-9b82-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/33b06c22-8065-11e2-9b82-52540035b04c</a></p> <p><a href="https://maloo.whamcloud.com/sub_tests/9ccebafe-807b-11e2-b777-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/9ccebafe-807b-11e2-b777-52540035b04c</a></p> <p>I think the underlying problem is that multiop as written can block on not just truncate lock, but also at group lock getting time it can block on cache flush.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> dd <span class="code-keyword">if</span>=/dev/zero of=$DIR/$tfile bs=1M count=10 || error $MULTIOP $DIR/$tfile OG${gid}T${sz}g${gid}c &amp; MULTIPID=$! sleep 2 </pre> </div></div> <p>Co perhaps we need to add sync or other way of cache flush after dd before doing multiop.</p> LU-2877

sanity test_34h failed Multiop blocked on ftruncate, pid=

Bug Critical Resolved Fixed Oleg Drokin Oleg Drokin mq313 Tue, 26 Feb 2013 22:24:12 +0000 Wed, 1 May 2013 06:45:12 +0000 Wed, 1 May 2013 06:45:12 +0000 Lustre 2.4.1 Lustre 2.4.0 0 4 <p>My idea of a patch is <a href="http://review.whamcloud.com/#change,5541" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5541</a></p> <p>Landed for 2.4</p> <p>My test at <a href="https://maloo.whamcloud.com/test_sets/0be54d54-8626-11e2-b472-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0be54d54-8626-11e2-b472-52540035b04c</a> still fails with Oleg's patch landed.</p> <p>This is from <a href="http://review.whamcloud.com/5470" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5470</a>, which has parent commit hash b7f949e04bbd4533316f0ca09b4b7d4f1765eca1, which is commit one after Oleg's patch commit hash 8fcac3f7a25c2d97974d292830dabe1611274085.</p> <p>There was another patch for a similar issue from Johann that also changed multiop, but I can no longer find it. Johann, do you remember?</p> <p>Also it's strange that sync was not enough, since there should be nothing to flush left.</p> <p>It was <a href="http://review.whamcloud.com/#change,5558" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5558</a> which also uses multiop_bg_pause. Not sure it really makes a difference ...</p> <p>That said, this latest report is with ldiskfs while all the previous ones i looked at were with ZFS ...</p> <p>On 2nd thought, i think multiop_bg_pause is not suitable here. We probably need to look at the logs to understand where we are slow/stuck.</p> <p>I wonder if it's a timing issue left - multiop starts late and has no chance to terminate by the time we check for it? Or I guess truncate serve-side mght be taking longer time under disk load?</p> <p>Ok, another attempt at fixing: <a href="http://review.whamcloud.com/5882" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5882</a></p> <p>Sanity test_32h hasn't failed in over a month, even before the most recent patch landed...</p> Development Rank 1|hzvjxj: Rank (Obsolete) 6951 Severity