https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1458 Lustre <p>This issue was created by maloo for yujian &lt;yujian@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/eb3d7ed4-ab13-11e1-8e7f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/eb3d7ed4-ab13-11e1-8e7f-52540035b04c</a>.</p> <p>The sub-test test_2b failed with the following error:</p> <blockquote> <p>Only in /mnt/lustre/d0.lustre-rsync-test/d2/clients/client1/~dmtmp/PM: PMD394.TMP<br/> lustre-rsync-test test_2b: @@@@@@ FAIL: Failure in replication; differences found. <br/> test failed to respond and timed out</p></blockquote> <p>Info required for matching: lustre-rsync-test 2b</p> LU-1458

lustre-rsync-test test_2b: old lustre_rsync does not work with new llog_changelog_ext_rec remove changelog

Bug Major Resolved Fixed Zhenyu Xu Maloo yuc2 Thu, 31 May 2012 07:40:44 +0000 Tue, 18 Mar 2014 19:07:00 +0000 Tue, 18 Mar 2014 19:07:00 +0000 Lustre 2.3.0 Lustre 2.1.2 Lustre 2.4.1 Lustre 2.5.0 Lustre 2.5.1 Lustre 2.4.0 Lustre 2.6.0 0 11 <p>Bobijam</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>I've tried on my VM machine, cannot hit the issue. </p> <p>ps, The test process is:</p> <p>1. run dbench on lustre direcotry.<br/> 2. use lustre_rsync to replicate lustre directory contents to another destination dir.<br/> 3. check whether source lustre directory differs from destination dir.</p> <p>This auto test case shows that /mnt/lustre/d0.lustre-rsync-test/d2/clients/client1/~dmtmp/PM/PMD394.TMP differs from its lustre_rsync-ed destination directory /tmp/target.</p> <p>I am not sure if this is the same issue:<a href="https://maloo.whamcloud.com/test_sets/81757bd0-ad72-11e1-8152-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/81757bd0-ad72-11e1-8152-52540035b04c</a></p> <p>client: 2.1.1-rhel6<br/> server: lustre-master-tag-2.2.54-rhel6</p> <p>Sarah,</p> <p>it's a different issue, and I've created a ticket for it (<a href="https://jira.whamcloud.com/browse/LU-1483" title="Test failure on test suite lustre-rsync-test" class="issue-link" data-issue-key="LU-1483"><del>LU-1483</del></a>)</p> <p><a href="https://maloo.whamcloud.com/test_sets/70d74642-bc21-11e1-8a1f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/70d74642-bc21-11e1-8a1f-52540035b04c</a></p> <p>could possibly related.</p> <p>Bobi, so the problem is not in the file content difference.</p> <p>The problem is this file only exists in the source dir, so it was not copied to target dir at all.<br/> So we need to find out why did that happen.</p> <p>I'll upload a debug patch to dump the changelog in plain text.</p> <p>When encounters another hit, would it possible to upload the changelog file as well? It contains all changelog records which lustre_rsync uses to replicate lustre source dir. </p> <p>debug improvement patch tracking at <a href="http://review.whamcloud.com/3551" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3551</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>patch description</b></div><div class="preformattedContent panelContent"> <pre>LU-1458 test: dump changelog for lustre-rsync-test Dump plain text format changelog records for failed lustre-rsync-test test case to help debugging. </pre> </div></div> <p>Diagnostic patch landed to master so should be in the next tag.</p> <p>Sarah</p> <p>Have you been able to test whether this issue still occurs since the diagnostic patch was landed or have you been blocked in doing so by another issue?</p> <p>Peter</p> <p>Hi Peter, due to TT-832, I cannot provision 2.1.1 client to verity this.</p> <p>Hit the similar issue with client running 2.1.1, server running lustre-master-tag2.2.93. The debug patch is for master only while this error was seen in interop testing which actually use the script on client. I am trying to port the changes to 2.1.1 and rerun the test.</p> <p>unfortunately this is the report without debug patch <br/> <a href="https://maloo.whamcloud.com/test_sets/da526672-ee0f-11e1-b95b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/da526672-ee0f-11e1-b95b-52540035b04c</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/0d1e405e-ee19-11e1-8649-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0d1e405e-ee19-11e1-8649-52540035b04c</a><br/> The attached are logs including changelog</p> <p>Sorry Sarah, please try <a href="http://review.whamcloud.com/3795" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3795</a> and reproduce it.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>patch description</b></div><div class="preformattedContent panelContent"> <pre> LU-1458 test: enable lustre_rsync debug log dump Make lustre_rsync dump its debug log to help debugging. </pre> </div></div> <p><a href="https://maloo.whamcloud.com/test_sets/fdfec3da-f08b-11e1-8816-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fdfec3da-f08b-11e1-8816-52540035b04c</a></p> <p>Sarah, </p> <p>The new patch changes lustre/utils/lustre_rsync.c, so we need deploy new images so that lustre_rsync can support this new -D option</p> <p>Bobi, the new build failed: <a href="http://build.whamcloud.com/job/lustre-reviews/8680/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/8680/</a></p> <p>done the rebuild</p> <blockquote> <p>Sarah,</p> <p>The new patch changes lustre/utils/lustre_rsync.c, so we need deploy new images so that lustre_rsync can support this new -D option</p></blockquote> <p>The patch is for master while this error occurs during interop testing between master and 2.1.x. I can manually port the script changes to 2.1.x but not the lustre_rsync.c. Could you please change that on 2.1.x so I can have a review build to test? </p> <p>b2_1 patch tracking at <a href="http://review.whamcloud.com/3822" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3822</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/5f1d7e18-f2e4-11e1-b39f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/5f1d7e18-f2e4-11e1-b39f-52540035b04c</a></p> <p>changelog</p> <p>from test_1.changelog</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>8 08RNMFM 17:53:31.834990148 2012.08.30 0x0 t=[0:0x0:0x0] p=[0x200000400:0x4:0x0] 9 01CREAT 17:53:31.838991641 2012.08.30 0x0 t=[0x200000400:0x9:0x0] p=[0x200000400:0x3:0x0] file4 </pre> </div></div> <p>and from lrsync_log.client_1.log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>***** Start 8 RNMFM (8) [0:0x0:0x0] [0x200000400:0x4:0x0] ***** move: /tmp/target/d0.lustre-rsync-test/d1/d2/ [to] /tmp/target/d0.lustre-rsync-test/d1/d1/file4 rc1=0, errno=95 move: /tmp/target2/d0.lustre-rsync-test/d1/d2/ [to] /tmp/target2/d0.lustre-rsync-test/d1/d1/file4 rc1=0, errno=95 ##### End 8 RNMFM (8) [0:0x0:0x0] [0x200000400:0x4:0x0] rc=0 ##### </pre> </div></div> <p>and the test_log error shows</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Only in /tmp/target/d0.lustre-rsync-test/d1/d1: file4 Only in /mnt/lustre/d0.lustre-rsync-test/d1: d2 </pre> </div></div> <p>the error must happen in lr_move(), lustre_rsync does not handle rename properly, still investigating.</p> <p>I did a rename operation in master branch, the changelog shows</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>19 08RENME 06:30:12.444665393 2012.08.31 0x0 t=[0:0x0:0x0] p=[0x200000400:0xc:0x0] file4 s=[0x200000400:0xd:0x0] sp=[0x200000400:0xb:0x0] d2 </pre> </div></div> <p>Sarah, what's the server version your test? I think the client is b2_1.</p> <p>I guess it's related to <a href="http://review.whamcloud.com/2577" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2577</a>, old lustre_rsync does not work with newer MDS server with regard to rename operation.</p> <p>b2_2 port of review#2577 tracking at <a href="http://review.whamcloud.com/3834" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3834</a><br/> b2_1 port of review#2577 tracking at <a href="http://review.whamcloud.com/3835" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3835</a><br/> b2_3 already has this change.</p> <p>server uses build 8694 from this review <a href="http://review.whamcloud.com/#change,3795" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3795</a><br/> client uses build 8726 from this review <a href="http://review.whamcloud.com/#change,3822" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3822</a></p> <p>Lustre client build: <a href="http://build.whamcloud.com/job/lustre-b2_1/121" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_1/121</a><br/> Lustre server build: <a href="http://build.whamcloud.com/job/lustre-b2_3/19" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/19</a><br/> Distro/Arch: RHEL6.3/x86_64</p> <p>lustre-rsync-test failed: <a href="https://maloo.whamcloud.com/test_sets/7075cfac-008c-11e2-860a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7075cfac-008c-11e2-860a-52540035b04c</a></p> <p>yujian,</p> <p>b2_1 patch <a href="http://review.whamcloud.com/#change,3835" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3835</a> hasn't landed yet. So lustre_rsync still does not work with b2_3 server.</p> <p>Lustre Tag: v2_3_0_RC2<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_3/32" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/32</a><br/> Distro/Arch: RHEL6.3/x86_64(server), RHEL5.8/x86_64(client)</p> <p>This issue occurred again: <a href="https://maloo.whamcloud.com/test_sets/fa89fd64-12b4-11e2-a23c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fa89fd64-12b4-11e2-a23c-52540035b04c</a></p> <p>Bobi, could you please check the above report? The failure occurred on a non-interop environment.</p> <p>can you upload the $LOGDIR/${TESTSUITE}.test_2b.changelog? (it should be generated on checkdiff error)</p> <blockquote><p>can you upload the $LOGDIR/${TESTSUITE}.test_2b.changelog? (it should be generated on checkdiff error)</p></blockquote> <p>Attached. Please check. Thanks.</p> <p>hmm. there are 1511 records in the lustre-rsync-test.test_2b.changelog, and the test log shows that lustre_rsync consumes 1510 records</p> <blockquote> <p>Changelog records consumed: 1510<br/> Only in /mnt/lustre/d0.lustre-rsync-test/d2/clients/client0/~dmtmp/ACCESS: INV.PRN<br/> lustre-rsync-test test_2b: @@@@@@ FAIL: Failure in replication; differences found. </p></blockquote> <p>and the 1511th in the lustre-rsync-test.test_2b.changelog is just the creation of the INV.PRN file</p> <blockquote> <p>7426 01CREAT 20:46:40.187891001 2012.10.09 0x0 t=<span class="error">&#91;0x200000400:0xd91:0x0&#93;</span> p=<span class="error">&#91;0x200000400:0xaf7:0x0&#93;</span> INV.PRN</p></blockquote> <p>Might be some changelog read/write competetion here.</p> <p>WangDi, is there a way to make sure all changelog recoreds are synced on its dt object?</p> <p>RHEL6.3/x86_64 (2.3.0 Server + 2.1.4 RC1 Client):<br/> <a href="https://maloo.whamcloud.com/test_sets/398724be-4871-11e2-8cdc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/398724be-4871-11e2-8cdc-52540035b04c</a></p> <p>Fresh Master error with logs: <a href="https://maloo.whamcloud.com/test_sets/04386912-d54d-11e2-bcd8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/04386912-d54d-11e2-bcd8-52540035b04c</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test_2b Error: 'test failed to respond and timed out' Failure Rate: 22.00% of last 100 executions [all branches] </pre> </div></div> <p>There is plenty of this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Replication of operation failed(-17): 4123 CREAT (1) [0x200000bd0:0x767:0x0] [0x200000bd0:0x766:0x0] client.txt Replication of operation failed(-17): 4124 CREAT (1) [0x200000bd0:0x768:0x0] [0x200000bd0:0x766:0x0] dbench Replication of operation failed(-17): 4125 MKDIR (2) [0x200000bd0:0x769:0x0] [0x200000bd0:0x766:0x0] lib64 Replication of operation failed(-17): 4126 CREAT (1) [0x200000bd0:0x76a:0x0] [0x200000bd0:0x769:0x0] libpopt.so.0 Replication of operation failed(-17): 4129 CREAT (1) [0x200000bd0:0x76b:0x0] [0x200000bd0:0x769:0x0] libc.so.6 </pre> </div></div> <p> 17 is EEXIST.</p> <p>It is not clear if this is the same exact issue but it fails with the same errors. </p> <p>Another one: <a href="https://maloo.whamcloud.com/sub_tests/0530d228-d54d-11e2-bcd8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/0530d228-d54d-11e2-bcd8-52540035b04c</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/652535cc-ddfc-11e2-a20c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/652535cc-ddfc-11e2-a20c-52540035b04c</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> lustre-rsync-test test_2b: @@@@@@ FAIL: Failure in replication; differences found. </pre> </div></div> <p><a href="https://maloo.whamcloud.com/test_sets/d9ef75f0-e416-11e2-8f78-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d9ef75f0-e416-11e2-8f78-52540035b04c</a> :</p> <p>lustre-rsync-test test_2b: @@@@@@ FAIL: Failure in replication; differences found.</p> <p>another: <a href="https://maloo.whamcloud.com/test_sets/bcadb2aa-035a-11e3-9f24-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/bcadb2aa-035a-11e3-9f24-52540035b04c</a></p> <p>lustre-rsync-test test_2b: @@@@@@ FAIL: Failure in replication; differences found.</p> <p>another: <a href="https://maloo.whamcloud.com/test_sets/2207a0c8-0439-11e3-a8e9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2207a0c8-0439-11e3-a8e9-52540035b04c</a></p> <p>+1 at <a href="https://maloo.whamcloud.com/test_sets/35af817e-0f54-11e3-9bce-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/35af817e-0f54-11e3-9bce-52540035b04c</a><br/> and it shows the same difference of one entry between the number of Changelog entries reported during the test and in the content gathered for the test, and this for the CREATE action of the file finally reported as missing during the check.<br/> I wonder if it is a problem with the Changelog sync itself or why not some timing issue with the background dbench real stop before lustre_rsync start ?</p> <p>Lustre client: <a href="http://build.whamcloud.com/job/lustre-b2_4/44/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/44/</a> (2.4.1 RC1)<br/> Lustre server: <a href="http://build.whamcloud.com/job/lustre-b2_3/41/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/41/</a> (2.3.0)</p> <p>lustre-rsync-test test 2b failed:<br/> <a href="https://maloo.whamcloud.com/test_sets/59afba4a-1502-11e3-ba63-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/59afba4a-1502-11e3-ba63-52540035b04c</a></p> <p>another<br/> <a href="https://maloo.whamcloud.com/test_sets/7c10d8f4-1e02-11e3-b42b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7c10d8f4-1e02-11e3-b42b-52540035b04c</a></p> <p>The spate of ZFS failures seem to be related to dbench not being started at the beginning of test 2b within the given 20s. Here is a patch to wait longer if necessary:</p> <p><a href="http://review.whamcloud.com/7914" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7914</a></p> <p>It seems this bug has been subverted from its original purpose of tracking a 2.1/2.4 interop problem into something unrelated that also causes test_2b to fail (dbench not starting quickly enough). It would be better to fix that problem in a separate bug, so that when the patch lands that bug can be closed, and this one is not closed.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/59/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/59/</a><br/> Distro/Arch: RHEL6.4/x86_64</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/2375f856-5817-11e3-b8c3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2375f856-5817-11e3-b8c3-52540035b04c</a></p> <p>More instance on Lustre b2_4 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/ed719c42-6356-11e3-8c76-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ed719c42-6356-11e3-8c76-52540035b04c</a></p> <p>An instance on Lustre b2_5 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/40cbb89e-7696-11e3-8c14-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/40cbb89e-7696-11e3-8c14-52540035b04c</a></p> <p>an instance in master:<br/> <a href="https://maloo.whamcloud.com/test_sets/736be8cc-7dc2-11e3-bfda-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/736be8cc-7dc2-11e3-bfda-52540035b04c</a></p> <p>More instances on Lustre b2_5 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/2d4b5c08-89b9-11e3-ae0e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2d4b5c08-89b9-11e3-ae0e-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/eec3c6d4-96c2-11e3-b941-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/eec3c6d4-96c2-11e3-b941-52540035b04c</a></p> <p>+1 on b2_5 branch : <a href="https://maloo.whamcloud.com/test_sessions/26ab637c-9b91-11e3-95f0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/26ab637c-9b91-11e3-95f0-52540035b04c</a></p> <p>I'm using <a href="https://jira.whamcloud.com/browse/LU-4781" title="lustre-rsync-test test_2b: Replication of operation failed(-17)" class="issue-link" data-issue-key="LU-4781"><del>LU-4781</del></a> as a replacement for this bug, which I think was fixed long ago.</p> Related LU-1331 LU-4781 LU-1442 LU-1237 Development Rank 1|hzv3lz: Rank (Obsolete) 4107 Severity