https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6233 Lustre <p>recovery-small test 10d failed with error message 'file contents differ'. Results and logs are at <a href="https://testing.hpdd.intel.com/test_sets/48de3eb8-ade9-11e4-a0b6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/48de3eb8-ade9-11e4-a0b6-5254006e85c2</a> . </p> <p>From the client test log, the test output is as expected until:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>... ldlm.namespaces.scratch-OST0005-osc-ffff8807dc5d1000.early_lock_cancel=1 ldlm.namespaces.scratch-OST0005-osc-ffff88080bd5ac00.early_lock_cancel=1 Connected clients: c13 c12 c11 c13 cmp: /lustre/scratch/f10d.recovery-small: Cannot send after transport endpoint shutdown recovery-small test_10d: @@@@@@ FAIL: file contents differ </pre> </div></div> OpenSFS Cluster with two MDSs each with one MDT, three OSSs each with two OSTs and three clients running lustre-master tag 2.6.93 build 2835 LU-6233

recovery-small test_10d failed with 'file contents differ'

Bug Minor Closed Cannot Reproduce WC Triage James Nunez Wed, 11 Feb 2015 14:58:35 +0000 Fri, 20 Jan 2017 21:19:14 +0000 Fri, 20 Jan 2017 21:19:14 +0000 Lustre 2.7.0 0 3 <p>This test was added in <a href="http://review.whamcloud.com/11752" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11752</a> "<a href="https://jira.whamcloud.com/browse/LU-5581" title="blocking ast error handling lack eviction for a local errors and some remote." class="issue-link" data-issue-key="LU-5581"><del>LU-5581</del></a> ldlm: evict clients returning errors on ASTs". We need a debug patch to find out what is going wrong, and whether this has turned a corner error case into a serious problem. </p> <p>I've reproduced this issue with lustre-master tag 2.6.94 and captured logs with full debug from the two MDSs, test10d_mds01_log.txt and test10d_mds02_log.txt, and from the client running recovery-small, test10d_client_log.txt, attached here. </p> <p>I added cat of the files when this error is hit. You can see below that I can't read /lustre/scratch/f10d.recovery-small ($DIR/$tfile); I get "Cannot send after transport endpoint shutdown" error. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>... Connected clients: c13 c13 c12 c11 cmp: /lustre/scratch/f10d.recovery-small: Cannot send after transport endpoint shutdown cat /lustre/scratch/f10d.recovery-small: cat: /lustre/scratch/f10d.recovery-small: Cannot send after transport endpoint shutdown end /lustre/scratch/f10d.recovery-small cat /lustre/scratch2/f10d.recovery-small: , worldend /lustre/scratch2/f10d.recovery-small recovery-small test_10d: @@@@@@ FAIL: file contents differ </pre> </div></div> <p>I can reproduce this error about one in 10 times running recovery-small.</p> <p>I did a check and recovery-small 10d has passed about 250 times in a row on master.</p> Duplicate LU-6359 Related LU-5581 LU-7759 Development Rank 1|hzx66f: Rank (Obsolete) 17460 Severity