https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7653 Lustre <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== replay-single test 110f: DNE: create striped dir, fail MDT1/MDT2 == 04:57:43 (1452229063) Filesystem 1K-blocks Used Available Use% Mounted on fre1225@tcp:/lustre 1377952 68464 1236816 6% /mnt/lustre Filesystem 1K-blocks Used Available Use% Mounted on fre1225@tcp:/lustre 1377952 68464 1236816 6% /mnt/lustre Failing mds1 on fre1225 Stopping /mnt/mds1 (opts:) on fre1225 pdsh@fre1227: fre1225: ssh exited with exit code 1 Failing mds2 on fre1225 Stopping /mnt/mds2 (opts:) on fre1225 pdsh@fre1227: fre1225: ssh exited with exit code 1 reboot facets: mds1 Failover mds1 to fre1225 04:57:56 (1452229076) waiting for fre1225 network 900 secs ... 04:57:56 (1452229076) network interface is UP mount facets: mds1 Starting mds1: -o rw,user_xattr /dev/vdb /mnt/mds1 pdsh@fre1227: fre1225: ssh exited with exit code 1 pdsh@fre1227: fre1225: ssh exited with exit code 1 Started lustre-MDT0000 reboot facets: mds2 Failover mds2 to fre1225 04:58:07 (1452229087) waiting for fre1225 network 900 secs ... 04:58:07 (1452229087) network interface is UP mount facets: mds2 Starting mds2: -o rw,user_xattr /dev/vdc /mnt/mds2 pdsh@fre1227: fre1225: ssh exited with exit code 1 pdsh@fre1227: fre1225: ssh exited with exit code 1 Started lustre-MDT0001 fre1228: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec fre1227: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec fre1228: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec fre1227: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec /mnt/lustre/d110f.replay-single/striped_dir has type dir OK replay-single test_110f: @@@@@@ FAIL: 1 != 2 after recovery Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4767:error_noexit() = /usr/lib64/lustre/tests/test-framework.sh:4798:error() = /usr/lib64/lustre/tests/replay-single.sh:3600:check_striped_dir_110() = /usr/lib64/lustre/tests/replay-single.sh:3725:test_110f() = /usr/lib64/lustre/tests/test-framework.sh:5045:run_one() = /usr/lib64/lustre/tests/test-framework.sh:5082:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:4899:run_test() = /usr/lib64/lustre/tests/replay-single.sh:3731:main() Dumping lctl log to /tmp/test_logs/1452229057/replay-single.test_110f.*.1452229095.log fre1228: Warning: Permanently added 'fre1227,192.168.112.27' (RSA) to the list of known hosts. fre1225: Warning: Permanently added 'fre1227,192.168.112.27' (RSA) to the list of known hosts. fre1226: Warning: Permanently added 'fre1227,192.168.112.27' (RSA) to the list of known hosts. FAIL 110f (33s) </pre> </div></div> Configuration : 4 Node - ( 1 MDS/1 OSS/2 Clients)_dne_singlemds<br/> Release<br/> 2.6.32_431.29.2.el6_lustremaster_9267_2_g959f8f7 Build Date: Sat 02 Jan 2016 05:21:40 PM UTC<br/> Server 2.7.64<br/> Client 2.7.64 LU-7653

replay-single/test_110f test failed Lustre: DEBUG MARKER: replay-single test_110f: FAIL: 1 != 2 after recovery

Bug Minor Resolved Fixed Alexander Boyko parinay v kondekar Tue, 12 Jan 2016 04:34:16 +0000 Tue, 14 Apr 2020 14:28:13 +0000 Tue, 14 Apr 2020 14:28:13 +0000 Lustre 2.8.0 Lustre 2.14.0 0 6 <p>Parinay - We don't see this test fail in our testing. How often do you see this failure and does it fail consistently for you?</p> <p>James,<br/> Its very consistent. I am attaching PTLDEBUG=-1 logs here.</p> <p>Thanks.</p> <p>This issue is very easy to reproduce when mds1 &amp; mds2 setup on same node. Looks like it can be fixed just changing failover order. </p> <p>Thanks,<br/> YangSheng</p> <p>We reproduce this issue, and as Mr Yang said, this issue can be fixed just changing the script as follows:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">test_110f() { ... mkdir -p $DIR/$tdir replay_barrier mds1 replay_barrier mds2 $LFS mkdir -i1 -c$MDSCOUNT $DIR/$tdir/striped_dir * fail mds2,mds1* check_striped_dir_110 || error <span class="code-quote">"check striped_dir failed"</span> rm -rf $DIR/$tdir || error <span class="code-quote">"rmdir failed"</span> <span class="code-keyword">return</span> 0 } </pre> </div></div> <p>And Our Lustre version is 2.9. Can some one help about this issue ? </p> <p>Parinay Kondekar (parinay.kondekar@seagate.com) uploaded a new patch: <a href="https://review.whamcloud.com/27940" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27940</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7653" title="replay-single/test_110f test failed Lustre: DEBUG MARKER: replay-single test_110f: FAIL: 1 != 2 after recovery" class="issue-link" data-issue-key="LU-7653"><del>LU-7653</del></a> tests: replay-single/110f fails for mdts on same MDS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 10e2e6ab2a1c449ccd2cd5cacac1efcbefed29fb</p> <p>But, why this issue can be fixed just change the fail order of mds1 and mds2 ? Can someone explain this ?</p> <p>Hi, ZhangWei,</p> <p>As my understanding, getdirstripe will try to get status from mds1 in DNE. If mds1 finished failover but mds2 does not, then we may get a stripecount less than mdscount. So keep mds1 finished failover in last can fix this issue especially when mds1 &amp; mds2 setup on same node.</p> <p>Thanks,<br/> YangSheng</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/27940/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27940/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7653" title="replay-single/test_110f test failed Lustre: DEBUG MARKER: replay-single test_110f: FAIL: 1 != 2 after recovery" class="issue-link" data-issue-key="LU-7653"><del>LU-7653</del></a> tests: replay-single/110f fails for mdts on same MDS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c245e87fad249622b1974dd64f3e497653269ee6</p> <p>Landed to 2.10.</p> <p>This is actually landed to master for 2.11.0.<br/> (fixing the fixVersion)</p> <p>Still seeing this failure very frequently on Oleg's test system:</p> <p>Test session:<br/> <a href="http://testing.linuxhacker.ru:3333/lustre-reports/3524/testresults/replay-single-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/" class="external-link" target="_blank" rel="nofollow noopener">http://testing.linuxhacker.ru:3333/lustre-reports/3524/testresults/replay-single-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/</a></p> <p>Subtest log:<br/> <a href="http://testing.linuxhacker.ru:3333/lustre-reports/3524/testresults/replay-single-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/replay-single.test_110f.test_log.oleg264-client.log" class="external-link" target="_blank" rel="nofollow noopener">http://testing.linuxhacker.ru:3333/lustre-reports/3524/testresults/replay-single-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/replay-single.test_110f.test_log.oleg264-client.log</a></p> <p>Alexander Boyko (c17825@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/38137" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38137</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7653" title="replay-single/test_110f test failed Lustre: DEBUG MARKER: replay-single test_110f: FAIL: 1 != 2 after recovery" class="issue-link" data-issue-key="LU-7653"><del>LU-7653</del></a> lod: fix stripe allocation during recovery<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 990d9d8cf607b035d2b341588212b77faf99f309</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38137/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38137/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7653" title="replay-single/test_110f test failed Lustre: DEBUG MARKER: replay-single test_110f: FAIL: 1 != 2 after recovery" class="issue-link" data-issue-key="LU-7653"><del>LU-7653</del></a> lod: fix stripe allocation during recovery<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 858c0c6959c1319e83a18be5ef6cb50251542052</p> <p>Landed for 2.14</p> Development Rank 1|hzxxw7: Rank (Obsolete) 9223372036854775807 Severity