https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11873 Lustre <p>sanity test 801a failed in review-ldiskfs test session on master branch as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>trevis-34vm4: Fail to freeze barrier for lustre: Timer expired CMD: trevis-34vm4 /usr/sbin/lctl get_param -n version 2&gt;/dev/null || /usr/sbin/lctl lustre_build_version 2&gt;/dev/null || /usr/sbin/lctl --version 2&gt;/dev/null | cut -d' ' -f2 CMD: trevis-34vm4 /usr/sbin/lctl barrier_stat -s lustre sanity test_801a: @@@@@@ FAIL: (2) unexpected barrier status 'expired' </pre> </div></div> <p>Maloo reports:<br/> <a href="https://testing.whamcloud.com/test_sets/f49fde42-1b26-11e9-8388-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/f49fde42-1b26-11e9-8388-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/900f4ab4-1ac0-11e9-8388-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/900f4ab4-1ac0-11e9-8388-52540065bddc</a></p> LU-11873

sanity test_801a: FAIL: (2) unexpected barrier status 'expired'

Bug Minor Resolved Fixed Patrick Farrell Jian Yu Fri, 18 Jan 2019 16:41:04 +0000 Thu, 12 Sep 2019 03:59:41 +0000 Wed, 21 Aug 2019 12:03:52 +0000 Lustre 2.13.0 Lustre 2.12.1 Lustre 2.13.0 Lustre 2.12.3 0 7 <p>+1 on b2_12 <a href="https://testing.whamcloud.com/test_sets/e298fc1e-4ad4-11e9-92fe-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e298fc1e-4ad4-11e9-92fe-52540065bddc</a></p> <p>I see this issue with master on a local setup very frequently.</p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/9d49e0f0-9756-11e9-8262-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/9d49e0f0-9756-11e9-8262-52540065bddc</a></p> <p>Logs from <a href="https://testing.whamcloud.com/test_sets/c432303a-9988-11e9-af8b-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/c432303a-9988-11e9-af8b-52540065bddc</a><br/> (<a href="https://review.whamcloud.com/#/c/35352/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/35352/</a>)</p> <p>00000004:00080000:0.0:1561700209.521557:0:11001:0:(osp_sync.c:1615:osp_sync_add_commit_cb()) lustre-OST0000-osc-MDT0001: add commit cb at 12164268321642ns, next at 11075831108412ns, rc = 0</p> <p>12164268321642-11075831108412</p> <p>On the node where the barrier command is being done, we start setting the barrier:<br/> 20000000:00000001:0.0:1561700207.130890:0:28927:0:(mgs_barrier.c:306:mgs_barrier_freeze()) Process entered<br/> 20000000:00000001:0.0:1561700209.536789:0:28927:0:(mgs_barrier.c:128:mgs_barrier_glimpse_lock()) Process entered</p> <p>And it doesn't complete until almost 10 seconds later:<br/> 20000000:00000001:1.0:1561700218.029393:0:28927:0:(mgs_barrier.c:205:mgs_barrier_glimpse_lock()) Process leaving via out (rc=0 : 0 : 0x0)<br/> By which time our barrier has expired.</p> <p>Looking at the glimpses, this one:<br/> 00000100:00000040:0.0:1561700209.537058:0:28927:0:(lustre_net.h:2491:ptlrpc_rqphase_move()) @@@ move req "New" <del>&gt; "Rpc" req@ffff93ba50f0cd80 x1637548785242880/t0(0) o106</del>&gt;MGS@10.2.8.58@tcp:15/16 lens 368/224 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1<br/> is sent normally, nice and quick...</p> <p>But it doesn't finish until:<br/> 00000100:00000200:1.0:1561700218.029267:0:28927:0:(events.c:93:reply_in_callback()) @@@ type 6, status 0 req@ffff93ba50f0cd80 x1637548785242880/t0(0) o106-&gt;MGS@10.2.8.58@tcp:15/16 lens 368/224 e 1 to 0 dl 1561700241 ref 1 fl Rpc:RQ/0/ffffffff rc 0/-1</p> <p>Much later, after the barrier has expired.</p> <p>Looking at the node where this glimpse was sent, we can see it arriving, and then generating a sync operation as part of turning on the barrier:<br/> 00000100:00000200:1.0:1561700209.521162:0:24526:0:(service.c:2234:ptlrpc_server_handle_request()) got req 1637548785242880<br/> <span class="error">&#91;...&#93;</span><br/> 00000004:00080000:1.0:1561700209.521565:0:24526:0:(osp_sync.c:1615:osp_sync_add_commit_cb()) lustre-OST0004-osc-MDT0003: add commit cb at 12164268329849ns, next at 12152647346967ns, rc = 0<br/> 12164268329849ns-12152647346967ns --&gt; ~11 seconds</p> <p>And we see:<br/> 00080000:00000020:1.0:1561700209.521610:0:24526:0:(osd_handler.c:658:osd_sync()) syncing OSD osd-zfs</p> <p>And then, after a nice long wait:<br/> 00080000:00000020:0.0:1561700218.012522:0:24526:0:(osd_handler.c:660:osd_sync()) synced OSD osd-zfs<br/> 00000004:00000001:0.0:1561700218.012525:0:24526:0:(lod_dev.c:1583:lod_sync()) Process leaving (rc=0 : 0 : 0)<br/> 00200000:40000000:0.0:1561700218.012528:0:24526:0:(barrier.c:221:barrier_freeze()) lustre-MDT0003-osd: barrier freezing phase1 done.<br/> 00200000:00000001:0.0:1561700218.012530:0:24526:0:(barrier.c:226:barrier_freeze()) Process leaving (rc=0 : 0 : 0)<br/> 00200000:00000001:0.0:1561700218.012531:0:24526:0:(barrier.c:344:barrier_handler()) Process leaving via fini (rc=0 : 0 : 0x0)</p> <p>And then (finally) we reply.</p> <p>So the issue is the ZFS sync interval being around 10 seconds, which is the same as the length of our barrier. So if we get unlucky, we'll overrun it.</p> <p>It's probably enough to change the barrier length to 15 seconds.</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35361" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35361</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11873" title="sanity test_801a: FAIL: (2) unexpected barrier status &#39;expired&#39;" class="issue-link" data-issue-key="LU-11873"><del>LU-11873</del></a> tests: Increase barrier freeze time<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f34ecbc3b15c6897be545f8241bf0f626997f183</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35361/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35361/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11873" title="sanity test_801a: FAIL: (2) unexpected barrier status &#39;expired&#39;" class="issue-link" data-issue-key="LU-11873"><del>LU-11873</del></a> tests: Increase barrier freeze time<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 96771280b330af07781326ff8811facd1ca39deb</p> <p>Landed for 2.13</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35952" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35952</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11873" title="sanity test_801a: FAIL: (2) unexpected barrier status &#39;expired&#39;" class="issue-link" data-issue-key="LU-11873"><del>LU-11873</del></a> tests: Increase barrier freeze time<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: b35866abc21a854e75cde15a6562ee2004c64b20</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35952/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35952/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11873" title="sanity test_801a: FAIL: (2) unexpected barrier status &#39;expired&#39;" class="issue-link" data-issue-key="LU-11873"><del>LU-11873</del></a> tests: Increase barrier freeze time<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 3a164a8d81395092add1426549821159db2e33a7</p> Related Development Rank 1|i009xj: Rank (Obsolete) 9223372036854775807 Severity