https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-68 Lustre <p>+ su mpiuser sh -c "/opt/mpich/ch-p4/bin/mpirun -np 12 -machinefile /tmp/parallel-scale.machines<br/> /usr/lib64/lustre/tests/write_disjoint -f /mnt/lustre/d0.write_disjoint/file -n 10000 "<br/> <span class="error">&#91;0&#93;</span> MPI Abort by user Aborting program !<br/> loop 0: chunk_size 103399<br/> loop 544: chunk_size 113838, file size was 1366056<br/> rank 0, loop 545: invalid file size 528737 instead of 576804 = 48067 * 12<br/> <span class="error">&#91;0&#93;</span> Aborting program!<br/> p4_error: latest msg from perror: Resource temporarily unavailable</p> <p>Reproduced at Oracle and I also have seen similar failures locally.<br/> Could be related to the <a href="https://jira.whamcloud.com/browse/LU-67" title="write_disjoint: data corruption" class="issue-link" data-issue-key="LU-67"><del>LU-67</del></a> issue</p> LU-68

write_disjoint: invalid file size

Bug Blocker Resolved Fixed Oleg Drokin Oleg Drokin Wed, 9 Feb 2011 12:29:18 +0000 Tue, 29 Mar 2011 11:12:41 +0000 Tue, 29 Mar 2011 11:12:41 +0000 Lustre 2.0.0 Lustre 2.1.0 Lustre 2.1.0 0 2 <p>This looks very similar to <a href="https://bugzilla.lustre.org/show_bug.cgi?id=3523" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=3523</a>.</p> <p>I have a log for this one now from my local testing.<br/> This is a case of page not sent to the server, so it might cause this or <a href="https://jira.whamcloud.com/browse/LU-67" title="write_disjoint: data corruption" class="issue-link" data-issue-key="LU-67"><del>LU-67</del></a> depending on if the page did not make at the end of file or in the middle.<br/> Something to do with incorrect kms it seems, still digging.</p> <p>I think I definitely foudn what the problem is finally<br/> in osc_lock_detach<br/>                /* Update the kms. Need to loop all granted locks.<br/>                 * Not a problem for the client */<br/>                attr-&gt;cat_kms = ldlm_extent_shift_kms(dlmlock, old_kms);<br/>                unlock_res_and_lock(dlmlock);<br/> &lt;&lt;HERE&gt;&gt;        <br/>                cl_object_attr_lock(obj);<br/>                cl_object_attr_set(env, obj, attr, CAT_KMS);<br/>                cl_object_attr_unlock(obj);</p> <p>for the discussed case the ldlm_shift_kms found an existing lock with bigger offset and returned old kms. As soon as we unlock, in comes another thread and updates kms (in our case it is write updating size in commit_write), then we proceed to write stale kms in the original thread and as a result the last page of the write is not reflected in kms and is lost.</p> <p>the problem did not happen in 1.8 because there the ldlm_extent_shift_kms was called under lov lock, but not anymore.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/reviews-centos5/448/" class="external-link" target="_blank" rel="nofollow noopener">reviews-centos5 #448</a><br/> <a href="https://jira.whamcloud.com/browse/LU-68" title="write_disjoint: invalid file size" class="issue-link" data-issue-key="LU-68"><del>LU-68</del></a> Fix a race between lock cancel and write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-release.git&amp;a=commit&amp;h=186df50693a7a0fd9e20b4ac0ac08d523f5473be" class="external-link" target="_blank" rel="nofollow noopener">186df50693a7a0fd9e20b4ac0ac08d523f5473be</a><br/> Files : </p> <ul> <li>lustre/osc/osc_lock.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master-centos5/151/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master-centos5 #151</a><br/> <a href="https://jira.whamcloud.com/browse/LU-68" title="write_disjoint: invalid file size" class="issue-link" data-issue-key="LU-68"><del>LU-68</del></a> Fix a race between lock cancel and write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-release.git&amp;a=commit&amp;h=d2dbff42e78d7ebca4db534df7e1c19f6b674a22" class="external-link" target="_blank" rel="nofollow noopener">d2dbff42e78d7ebca4db534df7e1c19f6b674a22</a><br/> Files : </p> <ul> <li>lustre/osc/osc_lock.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/reviews-rhel6/33/" class="external-link" target="_blank" rel="nofollow noopener">reviews-rhel6 #33</a><br/> <a href="https://jira.whamcloud.com/browse/LU-68" title="write_disjoint: invalid file size" class="issue-link" data-issue-key="LU-68"><del>LU-68</del></a> Fix a race between lock cancel and write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-dev.git&amp;a=commit&amp;h=d2dbff42e78d7ebca4db534df7e1c19f6b674a22" class="external-link" target="_blank" rel="nofollow noopener">d2dbff42e78d7ebca4db534df7e1c19f6b674a22</a><br/> Files : </p> <ul> <li>lustre/osc/osc_lock.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/reviews-centos5/483/" class="external-link" target="_blank" rel="nofollow noopener">reviews-centos5 #483</a><br/> <a href="https://jira.whamcloud.com/browse/LU-68" title="write_disjoint: invalid file size" class="issue-link" data-issue-key="LU-68"><del>LU-68</del></a> Fix a race between lock cancel and write</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-release.git&amp;a=commit&amp;h=d2dbff42e78d7ebca4db534df7e1c19f6b674a22" class="external-link" target="_blank" rel="nofollow noopener">d2dbff42e78d7ebca4db534df7e1c19f6b674a22</a><br/> Files : </p> <ul> <li>lustre/osc/osc_lock.c</li> </ul> <p>James</p> <p>When do you think that you might be able to try out your reproducer with the latest code?</p> <p>Please advise</p> <p>Peter</p> <p>Believed resolved. ORNL will reopen or open a new ticket if their reproducer still has issues</p> Bugzilla ID 23175.0 Development Rank 1|hzv9n3: Rank (Obsolete) 5096 Severity