https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4781 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;andreas.dilger@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/7edd3618-90d0-11e3-91ee-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/7edd3618-90d0-11e3-91ee-52540035b04c</a>.</p> <p>The sub-test test_2b failed with the following error in the test output:</p> <blockquote> <p>Replication of operation failed(-17): 4274 CREAT (1) <span class="error">&#91;0x400000400:0x7f4:0x0&#93;</span> <span class="error">&#91;0x400000400:0x7f3:0x0&#93;</span> SMALL.FIL<br/> Replication of operation failed(-17): 4275 CREAT (1) <span class="error">&#91;0x400000400:0x7f5:0x0&#93;</span> <span class="error">&#91;0x400000400:0x7ef:0x0&#93;</span> LARGE.FIL<br/> Replication of operation failed(-17): 4276 CREAT (1) <span class="error">&#91;0x400000400:0x7f6:0x0&#93;</span> <span class="error">&#91;0x400000400:0x7f3:0x0&#93;</span> MEDIUM.FIL<br/> Replication of operation failed(-17): 4277 CREAT (1) <span class="error">&#91;0x400000400:0x7f7:0x0&#93;</span> <span class="error">&#91;0x400000400:0x7f3:0x0&#93;</span> LARGE.FIL</p></blockquote> <p>Info required for matching: lustre-rsync-test 2b</p> LU-4781

lustre-rsync-test test_2b: Replication of operation failed(-17)

Bug Minor Resolved Fixed John Hammond Maloo Tue, 18 Mar 2014 19:03:18 +0000 Fri, 28 Oct 2016 16:59:55 +0000 Tue, 14 Jun 2016 22:49:50 +0000 Lustre 2.6.0 Lustre 2.5.3 Lustre 2.9.0 Lustre 2.9.0 0 15 <p>Lustre build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/80/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_5/80/</a><br/> Distro/Arch: RHEL6.5/x86_64<br/> FSTYPE=ldiskfs</p> <p>lustre-rsync-test test 2b failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Starting replication Replication of operation failed(-17): 3808 CREAT (1) [0x200000400:0x6de:0x0] [0x200000400:0x6dd:0x0] client.txt Replication of operation failed(-17): 3809 CREAT (1) [0x200000400:0x6df:0x0] [0x200000400:0x6dd:0x0] dbench Replication of operation failed(-17): 3810 MKDIR (2) [0x200000400:0x6e0:0x0] [0x200000400:0x6dd:0x0] lib64 Replication of operation failed(-17): 3811 CREAT (1) [0x200000400:0x6e1:0x0] [0x200000400:0x6e0:0x0] libpopt.so.0 --------8&lt;-------- Replication of operation failed(-17): 3954 CREAT (1) [0x200000400:0x76b:0x0] [0x200000400:0x766:0x0] LARGE.FIL ioctl err -22: Invalid argument (22) fid2path error: (/mnt/lustre, [0x200000400:0x903:0x0]) 22 Invalid argument Replication of operation failed(-22): 4819 CREAT (1) [0x200000400:0x903:0x0] [0x200000400:0x70e:0x0] GRAPH1.CDR Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre Target: /tmp/target Target: /tmp/target2 Statuslog: /tmp/lustre_rsync.log Changelog registration: cl4 Starting changelog record: 0 Clear changelog after use: no Errors: 142 lustre_rsync took 8 seconds Changelog records consumed: 1515 Only in /mnt/lustre/d2b.lustre-rsync-test/clients/client1/~dmtmp/WORDPRO: NEWS1_1B.PRN lustre-rsync-test test_2b: @@@@@@ FAIL: Failure in replication; differences found. </pre> </div></div> <p>Maloo report: <a href="https://testing.hpdd.intel.com/test_sets/de4f4760-26aa-11e4-9de1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/de4f4760-26aa-11e4-9de1-5254006e85c2</a></p> <p>I've run into the same test failure on lustre-master tag 2.6.94 in review-dne-part-1. Logs are at <a href="https://testing.hpdd.intel.com/test_sessions/3d80b6cc-b898-11e4-84be-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/3d80b6cc-b898-11e4-84be-5254006e85c2</a> .</p> <p>Other review-dne-part-1 failrues:<br/> 2015-07-09 15:01:36 - <a href="https://testing.hpdd.intel.com/test_sets/a6bf7468-26cd-11e5-8cf5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a6bf7468-26cd-11e5-8cf5-5254006e85c2</a><br/> 2015-07-10 03:03:45 - <a href="https://testing.hpdd.intel.com/test_sets/4df16926-26c4-11e5-8cf5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4df16926-26c4-11e5-8cf5-5254006e85c2</a><br/> 2015-07-11 21:05:09 - <a href="https://testing.hpdd.intel.com/test_sets/8d39b85e-282c-11e5-b280-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8d39b85e-282c-11e5-b280-5254006e85c2</a></p> <p>Another instance on master (pre-2.8) during review-dne-part-1:<br/> 2015-08-21 17:55:32 - <a href="https://testing.hpdd.intel.com/test_sets/8973091c-483b-11e5-813b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8973091c-483b-11e5-813b-5254006e85c2</a><br/> 2015-08-24 11:09:22 - <a href="https://testing.hpdd.intel.com/test_sets/a1d65e12-4a5e-11e5-88e8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a1d65e12-4a5e-11e5-88e8-5254006e85c2</a><br/> 2015-08-26 21:21:35 - <a href="https://testing.hpdd.intel.com/test_sets/f981d950-4c45-11e5-8a6a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f981d950-4c45-11e5-8a6a-5254006e85c2</a></p> <p>Another failure on master:<br/> 2015-12-10 14:51:22 - <a href="https://testing.hpdd.intel.com/test_sets/5c0fb6a8-9f72-11e5-8c89-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5c0fb6a8-9f72-11e5-8c89-5254006e85c2</a></p> <p>I did the following experiments on master branch and hit the similar failures:<br/> 1) format and setup the Lustre filesystem<br/> 2) on MDS, register a changelog user for MDT:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl --device lustre-MDT0000 changelog_register lustre-MDT0000: Registered changelog userid 'cl1' </pre> </div></div> <p>3) on Client, copy /etc to /mnt/lustre:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># cp -a /etc /mnt/lustre/ </pre> </div></div> <p>4) on Client, format an ext4 filesystem as the target backup filesystem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mkfs.ext4 /dev/vda3 # mount -t ext4 /dev/vda3 /mnt/backup/ </pre> </div></div> <p>5) on Client, perform rsync to make the source and target filesystems identical:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># rsync -avh /mnt/lustre/ /mnt/backup/ sending incremental file list --------8&lt;-------- sent 27.47M bytes received 23.02K bytes 4.23M bytes/sec total size is 27.38M speedup is 1.00 </pre> </div></div> <p>6) on Client, perform lustre_rsync:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lustre_rsync --source=/mnt/lustre/ --target=/mnt/backup/ --mdt=lustre-MDT0000 --user=cl1 --statuslog lustre_rsync_statuslog --verbose Replication of operation failed(-17): 1 MKDIR (2) [0x200000401:0x1:0x0] [0x200000007:0x1:0x0] etc Replication of operation failed(-17): 2 CREAT (1) [0x200000401:0x2:0x0] [0x200000401:0x1:0x0] issue.net Replication of operation failed(-17): 4 CREAT (1) [0x200000401:0x3:0x0] [0x200000401:0x1:0x0] yum.conf Replication of operation failed(-17): 6 MKDIR (2) [0x200000401:0x4:0x0] [0x200000401:0x1:0x0] gnupg Replication of operation failed(-17): 9 MKDIR (2) [0x200000401:0x5:0x0] [0x200000401:0x1:0x0] X11 Replication of operation failed(-17): 10 CREAT (1) [0x200000401:0x6:0x0] [0x200000401:0x5:0x0] prefdm Replication of operation failed(-17): 12 MKDIR (2) [0x200000401:0x7:0x0] [0x200000401:0x5:0x0] fontpath.d Replication of operation failed(-17): 13 SLINK (4) [0x200000401:0x8:0x0] [0x200000401:0x7:0x0] fonts-default Replication of operation failed(-17): 15 SLINK (4) [0x200000401:0x9:0x0] [0x200000401:0x7:0x0] default-ghostscript --------8&lt;-------- Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre/ Target: /mnt/backup/ Statuslog: lustre_rsync_statuslog Changelog registration: cl1 Starting changelog record: 0 Errors: 1782 lustre_rsync took 14 seconds Changelog records consumed: 3735 </pre> </div></div> <p>It turns out registering the changelog user needs to be done after running rsync. I did the experiment again and succeeded in running lustre_rsync:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lustre_rsync --source=/mnt/lustre/ --target=/mnt/backup/ --mdt=lustre-MDT0000 --user=cl1 --statuslog lustre_rsync_statuslog --verbose Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre/ Target: /mnt/backup/ Statuslog: lustre_rsync_statuslog Changelog registration: cl1 Starting changelog record: 0 Errors: 0 lustre_rsync took 0 seconds Changelog records consumed: 0 </pre> </div></div> <p>I think that this is because we use SIGSTOP and SIGCONT to control dbench.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> echo Stopping dbench $KILL -SIGSTOP $child_pid echo Starting replication $LRSYNC -l $LREPL_LOG -D $LRSYNC_LOG check_diff $DIR/$tdir $TGT/$tdir </pre> </div></div> <p>Even though <tt>kill()</tt> has returned the SIGSTOP may still be pending (for example, if the the dbench task is executing the the kernel, doing I/O). We should sleep for some time after sending SIGSTOP or even better wait until all dbench tasks have a stopped state.</p> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/20471" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20471</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4781" title="lustre-rsync-test test_2b: Replication of operation failed(-17)" class="issue-link" data-issue-key="LU-4781"><del>LU-4781</del></a> test: wait for dbench to stop<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d5fe8aeb0b6ea00165ddc98e13c5fb270fad8e7d</p> <p>Andreas Dilger (andreas.dilger@intel.com) merged in patch <a href="http://review.whamcloud.com/20471/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20471/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4781" title="lustre-rsync-test test_2b: Replication of operation failed(-17)" class="issue-link" data-issue-key="LU-4781"><del>LU-4781</del></a> test: wait for dbench to stop<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f7452bafb79d94c88b5fb390e431b06e79325f99</p> <p>lustre-rsyn-test test 2b is still failing even after patch 20471 landed to master. Here are some recent failures:<br/> 2016-06-05 - <a href="https://testing.hpdd.intel.com/test_sets/f3692b30-2b07-11e6-80b9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f3692b30-2b07-11e6-80b9-5254006e85c2</a><br/> 2016-06-05 - <a href="https://testing.hpdd.intel.com/test_sets/0485f5aa-2b5e-11e6-80b9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0485f5aa-2b5e-11e6-80b9-5254006e85c2</a><br/> 2016-06-05 - <a href="https://testing.hpdd.intel.com/test_sets/9874c1e0-2b60-11e6-80b9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9874c1e0-2b60-11e6-80b9-5254006e85c2</a><br/> 2016-06-06 - <a href="https://testing.hpdd.intel.com/test_sets/36032394-2ba6-11e6-acf3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/36032394-2ba6-11e6-acf3-5254006e85c2</a></p> <p>I think it is strange that the error -17 = <tt>-EEXIST</tt> is the only one being hit. Is the MDS not properly registering or processing unlink records? Is lustre_rsync returning an error for <tt>CREAT</tt> records (e.g. using <tt>O_EXCL</tt>) when they should be allowed to open(O_CREAT) files that already exist? Is lustre_rsync not applying the CREAT records in the right parent directory?</p> <p>Is it worthwhile to submit a test patch to collect the full changelog from the dbench run so that it is possible to trace the complete lifetime of a parent directory and file so that this problem can be debugged more effectively?</p> <p>Looking at the lustre_rsync log it appears that we are processing the ChangeLog entries twice?<br/> <a href="https://testing.hpdd.intel.com/sub_tests/1a5aee48-2be6-11e6-a0ce-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/1a5aee48-2be6-11e6-a0ce-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_logs/1a89dee2-2be6-11e6-a0ce-5254006e85c2/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_logs/1a89dee2-2be6-11e6-a0ce-5254006e85c2/show_text</a><br/> <a href="https://testing.hpdd.intel.com/test_logs/1aa09a56-2be6-11e6-a0ce-5254006e85c2/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_logs/1aa09a56-2be6-11e6-a0ce-5254006e85c2/show_text</a></p> <p>For example, the file "STUDENTS.TV" FID <span class="error">&#91;0x200006990:0x414:0x0&#93;</span> is created in record <tt>5056</tt> but that record appears twice in the log and the second time produces the <tt>-EEXIST</tt> error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>**** Start 5056 CREAT (1) [0x200006990:0x414:0x0] [0x200006990:0x3f0:0x0] STUDENTS.TV ***** dest = d2b.lustre-rsync-test/clients/client1/~dmtmp/PARADOX/STUDENTS.TV; savedpath = d2b.lustre-rsync-test/clients/client1/~dmtmp/PARADOX/STUDENTS.TV mkfile(1) /tmp/target/d2b.lustre-rsync-test/clients/client1/~dmtmp/PARADOX/STUDENTS.TV Syncing data and attributes [0x200006990:0x414:0x0] llistxattr(/mnt/lustre/.lustre/fid/[0x200006990:0x414:0x0],(nil)) returned 64, errno=0 llistxattr /mnt/lustre/.lustre/fid/[0x200006990:0x414:0x0] returned 64, errno=0 (trusted.version,10469904) rc=0x8 lsetxattr(), rc=0, errno=0 (trusted.link,10469904) rc=0x35 lsetxattr(), rc=0, errno=0 (trusted.lov,10469904) rc=0x38 lsetxattr(), rc=0, errno=0 (trusted.lma,10469904) rc=0x18 lsetxattr(), rc=0, errno=0 (lustre.lov,10469904) rc=0x38 lsetxattr(), rc=-1, errno=95 setxattr: /mnt/lustre/.lustre/fid/[0x200006990:0x414:0x0] /tmp/target/d2b.lustre-rsync-test/clients/client1/~dmtmp/PARADOX/STUDENTS.TV mkfile(1) /tmp/target2/d2b.lustre-rsync-test/clients/client1/~dmtmp/PARADOX/STUDENTS.TV Syncing data and attributes [0x200006990:0x414:0x0] llistxattr(/mnt/lustre/.lustre/fid/[0x200006990:0x414:0x0],0x9fb200) returned 64, errno=0 (trusted.version,10469904) rc=0x8 lsetxattr(), rc=0, errno=0 (trusted.link,10469904) rc=0x35 lsetxattr(), rc=0, errno=0 (trusted.lov,10469904) rc=0x38 lsetxattr(), rc=0, errno=0 (trusted.lma,10469904) rc=0x18 lsetxattr(), rc=0, errno=0 (lustre.lov,10469904) rc=0x38 lsetxattr(), rc=-1, errno=95 setxattr: /mnt/lustre/.lustre/fid/[0x200006990:0x414:0x0] /tmp/target2/d2b.lustre-rsync-test/clients/client1/~dmtmp/PARADOX/STUDENTS.TV ##### End 5056 CREAT (1) [0x200006990:0x414:0x0] [0x200006990:0x3f0:0x0] STUDENTS.TV rc=0 ##### : : ***** Start 5058 CLOSE (11) [0x200006990:0x414:0x0] [0:0x0:0x0] ***** ##### End 5058 CLOSE (11) [0x200006990:0x414:0x0] [0:0x0:0x0] rc=0 ##### : : : ***** Start 5056 CREAT (1) [0x200006990:0x414:0x0] [0x200006990:0x3f0:0x0] STUDENTS.TV ***** dest = d2b.lustre-rsync-test/clients/client1/~dmtmp/PARADOX/STUDENTS.TV; savedpath = d2b.lustre-rsync-test/clients/client1/~dmtmp/PARADOX/STUDENTS.TV mkfile(1) /tmp/target/d2b.lustre-rsync-test/clients/client1/~dmtmp/PARADOX/STUDENTS.TV mkfile(1) /tmp/target2/d2b.lustre-rsync-test/clients/client1/~dmtmp/PARADOX/STUDENTS.TV ##### End 5056 CREAT (1) [0x200006990:0x414:0x0] [0x200006990:0x3f0:0x0] STUDENTS.TV rc=-17 ##### ***** Start 5058 CLOSE (11) [0x200006990:0x414:0x0] [0:0x0:0x0] ***** ##### End 5058 CLOSE (11) [0x200006990:0x414:0x0] [0:0x0:0x0] rc=0 ##### </pre> </div></div> <p>It appears that the log processing restarts at some point, but I don't know what the cause is. It might be ChangeLog wrapping in the kernel?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>: : ***** Start 5519 CLOSE (11) [0x200006990:0x460:0x0] [0:0x0:0x0] ***** ##### End 5519 CLOSE (11) [0x200006990:0x460:0x0] [0:0x0:0x0] rc=0 ##### ***** Start 5520 CLOSE (11) [0x200006990:0x460:0x0] [0:0x0:0x0] ***** ##### End 5520 CLOSE (11) [0x200006990:0x460:0x0] [0:0x0:0x0] rc=0 ##### ***** Start 4823 CREAT (1) [0x200006990:0x397:0x0] [0x200006990:0x395:0x0] client.txt ***** dest = d2b.lustre-rsync-test/client.txt; savedpath = d2b.lustre-rsync-test/client.txt mkfile(1) /tmp/target/d2b.lustre-rsync-test/client.txt mkfile(1) /tmp/target2/d2b.lustre-rsync-test/client.txt ##### End 4823 CREAT (1) [0x200006990:0x397:0x0] [0x200006990:0x395:0x0] client.txt rc=-17 ##### ***** Start 4824 CLOSE (11) [0x200006990:0x397:0x0] [0:0x0:0x0] ***** ##### End 4824 CLOSE (11) [0x200006990:0x397:0x0] [0:0x0:0x0] rc=0 ##### : : </pre> </div></div> <p>Hi John,</p> <p>Can you have another look at this one? It seems even with the landing of <a href="http://review.whamcloud.com/#/c/20471/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/20471/</a> that the problem is still occurring.</p> <p>Thanks.<br/> Joe</p> <p>&gt; I think it is strange that the error -17 = -EEXIST is the only one being hit. Is the MDS not properly registering or processing unlink records? Is lustre_rsync returning an error for CREAT records (e.g. using O_EXCL) when they should be allowed to open(O_CREAT) files that already exist? Is lustre_rsync not applying the CREAT records in the right parent directory?</p> <p>It uses <tt>mknod()</tt>. The <tt>-EEXIST</tt> errors are also seen on successful runs. But it's conceivable that there is a race where the unhandled <tt>-EEXIST</tt> causes the files to end up with different contents. I'll push a small patch.</p> <p>I think the more important problem is that it appears lustre_rsync is processing the same ChangeLog records twice. Is that because lustre_rsync is processing the logs incorrectly, because the "clear logs before record X" request isn't handled or isn't persistent (ala <a href="https://jira.whamcloud.com/browse/LU-7428" title="conf-sanity test_84, replay-dual 0a: /dev/lvm-Role_MDS/P1 failed to initialize!" class="issue-link" data-issue-key="LU-7428"><del>LU-7428</del></a>) across restarts, or because of an internal ChangeLog bug that is returning bogus duplicate records due to ChangeLog index wrap, bad llog handling, etc?</p> <p>&gt; For example, the file "STUDENTS.TV" FID <span class="error">&#91;0x200006990:0x414:0x0&#93;</span> is created in record 5056 but that record appears twice in the log and the second time produces the -EEXIST error:</p> <p>This is because <tt>lustre_rsync</tt> is invoked with <tt>-c no</tt> which tells it not to clear the clangelog when done.</p> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/20649" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20649</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4781" title="lustre-rsync-test test_2b: Replication of operation failed(-17)" class="issue-link" data-issue-key="LU-4781"><del>LU-4781</del></a> utils: handle EEXIST in lustre_rsync<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bb459c9631361cb28bd8fe367de99603e3d67ff0</p> <p>Another failure instance on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/f7240550-308a-11e6-a0ce-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f7240550-308a-11e6-a0ce-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20649/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20649/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4781" title="lustre-rsync-test test_2b: Replication of operation failed(-17)" class="issue-link" data-issue-key="LU-4781"><del>LU-4781</del></a> utils: handle EEXIST in lustre_rsync<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2a55f34bd5514e905f6049f4e293352bd09ffe91</p> <p>2nd patch has landed to master for 2.9</p> <p>more on b2_8_fe:<br/> <a href="https://testing.hpdd.intel.com/test_sets/885f908c-7f8e-11e6-8a8c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/885f908c-7f8e-11e6-8a8c-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/25fa9bc8-7ec9-11e6-8afd-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/25fa9bc8-7ec9-11e6-8afd-5254006e85c2</a></p> Duplicate Related LU-2121 LU-4256 LU-1458 LU-6644 Development Rank 1|hzwhsf: Rank (Obsolete) 13146 Severity