https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1060 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/c95fc5c2-4c42-11e1-bd50-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c95fc5c2-4c42-11e1-bd50-5254004bbbd3</a>.</p> <p>The sub-test test_7c failed with the following error:</p> <blockquote> <p>Test 7c.2 failed</p></blockquote> <p>Info required for matching: replay-vbr 7c</p> LU-1060

Test failure on test suite replay-vbr, subtest test_7c

Bug Blocker Resolved Duplicate nasf Maloo Tue, 31 Jan 2012 15:53:35 +0000 Thu, 16 Feb 2012 16:15:13 +0000 Thu, 16 Feb 2012 16:15:13 +0000 Lustre 2.2.0 Lustre 2.1.1 Lustre 2.2.0 0 4 <p>dmesg and debug log from servers</p> <p>Fanyong</p> <p>Could you please look into this 2.2 blocker?</p> <p>Thanks</p> <p>Peter</p> <p>Lustre Tag: v2_1_1_0_RC1<br/> Distro/Arch: RHEL5.7 (server), SLES11SP1 (client)<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/37/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/37/</a><br/> Network: TCP</p> <p>The same issue occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/e2334988-5300-11e1-81a2-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/e2334988-5300-11e1-81a2-5254004bbbd3</a></p> <p>Lustre Clients:<br/> Tag: 2.1.0<br/> Distro/Arch: RHEL5/x86_64 (kernel version: 2.6.18-238.19.1.el5)<br/> Network: IB (in-kernel OFED)<br/> ENABLE_QUOTA=yes</p> <p>Lustre Servers:<br/> Tag: v2_1_1_0_RC1<br/> Distro/Arch: RHEL5/x86_64 (kernel version: 2.6.18-274.12.1.el5_lustre)<br/> Build: <a href="http://build.whamcloud.com/job/lustre-b2_1/37/arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/37/arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/</a><br/> Network: IB (in-kernel OFED)</p> <p>The same issue occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/68802da0-54bc-11e1-9fd4-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/68802da0-54bc-11e1-9fd4-5254004bbbd3</a></p> <p>There are some defects in current VBR implementation. For example in replay-vbr.sh test_7c</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">test_7c() { ... first=<span class="code-quote">"createmany -o $DIR/$tdir/$tfile- 2"</span> lost=<span class="code-quote">"rm $MOUNT2/$tdir/$tfile-0; mkdir $MOUNT2/$tdir/$tfile-0"</span> last=<span class="code-quote">"mv $DIR/$tdir/$tfile-1 $DIR/$tdir/$tfile-0"</span> test_7_cycle <span class="code-quote">"$first"</span> <span class="code-quote">"$lost"</span> <span class="code-quote">"$last"</span> || error <span class="code-quote">"Test 7c.2 failed"</span> ... } </pre> </div></div> <p>The operations sequence in test_7_cycle() is as following:</p> <p>(step 0) replay barrier<br/> (step 1) client1 "first"<br/> (step 2) client2 "lost"<br/> (step 3) client1 "last"<br/> (step 4) umount client2<br/> (step 5) MDS failover<br/> (step 6) client1 try to recover</p> <p>Since client2 umount before MDS failover, and client1's "last" operation depends on client2's "lost" operation, client1 is expected to fail to replay the "last" operation.</p> <p>But now we found client1 was not evicted after MDS failover. The reason is as following:</p> <p>The original $tfile-0 FID was "FID_001" when created by client1 step 1, then the client2 unlink $tfile-0, and mkdir with the same name, the new FID for $tfile-0 is "FID_002" by step 2. When client1 performed step 3, it used the $tfile-0's new "FID_002", and such FID also be used when client1 replayed "last" during MDS failover. But during the MDS failover, client2 missed, then nobody re-created $tfile-0 with new "FID_002", so when client1 tried to replay "last" during MDS failover, it could not find the target $tfile-0 with "FID_002". Under such case, client1's recovery should be regarded as failure, and client1 should be evicted. But in current implementation, during the VBR phase, only evict the client with VBR failure (for object version unmatched cases). test_7c failed for "-ENOENT", no chance to compare the versions yet, so client1 was not evicted.</p> <p>The simplest way to resolve the issue is to regard all target missed cases during recovery as VBR failure, then evict related client.</p> <p>The patch for above issue:</p> <p><a href="http://review.whamcloud.com/#change,2144" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2144</a></p> <p>VBR detects ENOENT cases already and fails if some object is missing. For such object VBR count its version as ENOENT_VERSION and compare it with version in replay, FID_002 in your example. So there must be version mismatch. If that doesn't work for some reason, we need to find that reason. Keep in mind that this worked well for a long time but failed in 2.1&lt;-&gt;2.1.55 case, so probably this is compatibility issue between lustre versions. First of all I'd check where that -ENOENT came from and why VBR checks missed it.</p> <p>Looking through server syslog:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Lustre: 31370:0:(client.c:2530:ptlrpc_replay_interpret()) @@@ Version mismatch during replay req@ffff88031a8b5000 x1392520277794082/t657129996304(657129996304) o-1-&gt;lustre-MDT0000_UUID@192.168.4.134@o2ib:12/10 lens 472/424 e 0 to 0 dl 1328013402 ref 2 fl Interpret:R/ffffffff/ffffffff rc -75/-1 Lustre: 31370:0:(client.c:2530:ptlrpc_replay_interpret()) Skipped 3 previous similar messages Lustre: 31370:0:(client.c:1778:ptlrpc_expire_one_request()) @@@ Request x1392520277794320 sent from lustre-MDT0000-mdc-ffff880307c27800 to NID 192.168.4.134@o2ib has timed out <span class="code-keyword">for</span> slow reply: [sent 1328013368] [real_sent 1328013368] [current 1328013402] [deadline 34s] [delay 0s] req@ffff880316c8f800 x1392520277794320/t0(0) o-1-&gt;lustre-MDT0000_UUID@192.168.4.134@o2ib:12/10 lens 192/192 e 0 to 1 dl 1328013402 ref 1 fl Rpc:X/ffffffff/ffffffff rc 0/-1 Lustre: 31370:0:(client.c:1778:ptlrpc_expire_one_request()) Skipped 37 previous similar messages Lustre: 31370:0:(<span class="code-keyword">import</span>.c:1160:completed_replay_interpret()) lustre-MDT0000-mdc-ffff880307c27800: version recovery fails, reconnecting LustreError: 167-0: This client was evicted by lustre-MDT0000; in progress operations using <span class="code-keyword">this</span> service will fail. </pre> </div></div> <p>That shows the VBR works fine, mismatch was detected and recovery fails.</p> <p>but a bit later:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Lustre: lustre-MDT0000-mdc-ffff88024b596400: Connection to service lustre-MDT0000 via nid 192.168.4.134@o2ib was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete. Lustre: Skipped 9 previous similar messages Lustre: 31370:0:(<span class="code-keyword">import</span>.c:852:ptlrpc_connect_interpret()) MGS@192.168.4.134@o2ib changed server handle from 0xa0636bc0947b83ea to 0xa0636bc0947b8a57 LustreError: 31370:0:(client.c:2573:ptlrpc_replay_interpret()) @@@ status -2, old was 0 req@ffff8802c7375800 x1392520277794470/t661424963601(661424963601) o-1-&gt;lustre-MDT0000_UUID@192.168.4.134@o2ib:12/10 lens 472/424 e 0 to 0 dl 1328013526 ref 2 fl Interpret:R/ffffffff/ffffffff rc -2/-1 Lustre: lustre-MDT0000-mdc-ffff88024b596400: Connection restored to service lustre-MDT0000 using nid 192.168.4.134@o2ib </pre> </div></div> <p>I have no idea what is that so far</p> <p>It is not interoperability issue, it can be reproduced against latest master branch by replay-vbr test_7c.</p> <p>this is result of <a href="https://jira.whamcloud.com/browse/LU-966" title="post-fsck MDS LBUG during recovery due to missing FID" class="issue-link" data-issue-key="LU-966"><del>LU-966</del></a> patch, unfortunately nobody noticed it ruins VBR recovery because mdt_object_find() may exits early without any VBR checking.</p> <p>Right, so to erase the side-affect of <a href="https://jira.whamcloud.com/browse/LU-966" title="post-fsck MDS LBUG during recovery due to missing FID" class="issue-link" data-issue-key="LU-966"><del>LU-966</del></a> patch, my patch works well. Otherwise, we have to fix up all related points one by one, and if someone add new points in the future, he/she has to consider again.</p> <p>So, what's your idea?</p> <p><a href="https://jira.whamcloud.com/browse/LU-1060" title="Test failure on test suite replay-vbr, subtest test_7c" class="issue-link" data-issue-key="LU-1060"><del>LU-1060</del></a> is caused by <a href="https://jira.whamcloud.com/browse/LU-966" title="post-fsck MDS LBUG during recovery due to missing FID" class="issue-link" data-issue-key="LU-966"><del>LU-966</del></a> improper fix.</p> <p>Right, your patch will work to cover some case but it is just fast fix to hide bad effects of previous wrong patch, that is the way we shouldn't go for sure. It hides side-effects but doesn't fix the root cause, moreover it doesn't fix broken VBR which can cause unneeded evictions after <a href="https://jira.whamcloud.com/browse/LU-966" title="post-fsck MDS LBUG during recovery due to missing FID" class="issue-link" data-issue-key="LU-966"><del>LU-966</del></a>. The right way will be step back to the point where all issues appeared - <a href="https://jira.whamcloud.com/browse/LU-966" title="post-fsck MDS LBUG during recovery due to missing FID" class="issue-link" data-issue-key="LU-966"><del>LU-966</del></a>.</p> <p>Basically we need to revert that patch and apply its first version - just replace assertions with error checks, it straight-forward and easy to follow. The idea to make that early in MDT was wrong and we missed that, any further attempts to fix that in MDT will cause more complexity there and more checks. I've made patch already:<br/> <a href="http://review.whamcloud.com/#change,2148" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2148</a></p> <p>Another my worry is about test set for master review testing, I don't get why it misses replay-vbr and runtests which are pretty good tests. <a href="https://jira.whamcloud.com/browse/LU-1060" title="Test failure on test suite replay-vbr, subtest test_7c" class="issue-link" data-issue-key="LU-1060"><del>LU-1060</del></a> appeared right after <a href="https://jira.whamcloud.com/browse/LU-966" title="post-fsck MDS LBUG during recovery due to missing FID" class="issue-link" data-issue-key="LU-966"><del>LU-966</del></a> landing and nobody noticed that. This is out of scope this bug though.</p> <p>Removing the patch for <a href="https://jira.whamcloud.com/browse/LU-966" title="post-fsck MDS LBUG during recovery due to missing FID" class="issue-link" data-issue-key="LU-966"><del>LU-966</del></a> directly may be not the best solution. If you do not like my patch for <a href="https://jira.whamcloud.com/browse/LU-1060" title="Test failure on test suite replay-vbr, subtest test_7c" class="issue-link" data-issue-key="LU-1060"><del>LU-1060</del></a>, we can fix it case by case. I think Bobijam is working on such patch to erase the side-effect of his <a href="https://jira.whamcloud.com/browse/LU-966" title="post-fsck MDS LBUG during recovery due to missing FID" class="issue-link" data-issue-key="LU-966"><del>LU-966</del></a> patch.</p> <p>the patch to let vbr version check replay non-exist object is posted at <a href="http://review.whamcloud.com/2149" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2149</a></p> <p>description: For replay cases, mdt_version_get_check will check non-exist mdt object and evict clients accordingly, but mdt_object_find will not set exp_vbr_failed and will not evict the faulty client.</p> <p>duplicate of lu-966</p> Related LU-966 Development Rank 1|hzvhfj: Rank (Obsolete) 6473 Severity