https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4597 Lustre <p>We have received reports of Lustre clients incorrectly reporting files with 0 length then on a second attempt the correct non-zero length will be reported. This is reminiscent of <a href="https://jira.whamcloud.com/browse/LU-274" title="Client delayed file status (cache meta-data) causing job failures" class="issue-link" data-issue-key="LU-274"><del>LU-274</del></a>. Before digging too far into this, I notice that the <a href="http://review.whamcloud.com/#/c/507/2" class="external-link" target="_blank" rel="nofollow noopener">fix</a> for <a href="https://jira.whamcloud.com/browse/LU-274" title="Client delayed file status (cache meta-data) causing job failures" class="issue-link" data-issue-key="LU-274"><del>LU-274</del></a> did not seem to survive the conversion from obdfilter to ofd. Do we need a similar fix in <tt>ofd_intent_policy()</tt>?</p> <p>Here is the reproducer reported by our user:</p> <ol> <li>On cslic3 using lscratch3, I created 20 files of size 1024 bytes</li> <li>Waited 27 minutes (I tried around 15 minutes and didn't see incorrect size zeros)</li> <li>On cslic8 listed the lscratch3 directory from step 1, I saw 3 files with size 0</li> <li>On cslic8 listed the lscratch3 directory again (immediately following step3) and all files listed as 1024 bytes</li> </ol> 2.4.0-19chaos clients and servers LU-4597

inconsistent file size

Bug Critical Resolved Fixed Niu Yawei Ned Bass mn4 Thu, 6 Feb 2014 23:01:23 +0000 Mon, 14 Apr 2014 20:44:45 +0000 Wed, 26 Feb 2014 23:18:09 +0000 Lustre 2.4.0 Lustre 2.6.0 Lustre 2.5.1 0 10 <p>Oh, I see <a href="https://jira.whamcloud.com/browse/LU-274" title="Client delayed file status (cache meta-data) causing job failures" class="issue-link" data-issue-key="LU-274"><del>LU-274</del></a> was fixed for b_2.x in <a href="http://review.whamcloud.com/583" class="external-link" target="_blank" rel="nofollow noopener">this patch</a>.</p> <p>Niu</p> <p>Could you please help Prakash with any questions he has about this work?</p> <p>Thanks</p> <p>Peter</p> <p>The fix of <a href="https://jira.whamcloud.com/browse/LU-274" title="Client delayed file status (cache meta-data) causing job failures" class="issue-link" data-issue-key="LU-274"><del>LU-274</del></a> is in ldlm_cb_interpret() of 2.4, so looks this is different problem than <a href="https://jira.whamcloud.com/browse/LU-274" title="Client delayed file status (cache meta-data) causing job failures" class="issue-link" data-issue-key="LU-274"><del>LU-274</del></a>.</p> <p>Ned, is it only seen on zfs or it's a common issue? Can this be easily reproduced by the procedures provided by you? Thank you.</p> <p>I believe we've seen it on both ZFS and ldiskfs, but I'll verify that next week. It's not easily reproducible. I've been running the procedure in a loop for over 24 hours and haven't reproduced it yet.</p> <p>Niu, our archival storage servers log an error message during file transfer if a file size changes after an initial scan, so we can use that as evidence of this bug. The logs show a sharp increase in file sizes changing from 0 after we updated our Lustre servers from 2.1 to 2.4.0-19chaos. This has affected all of our filesystems running both ZFS and ldiskfs. It has been observed from both 2.1 and 2.4 clients.</p> <p>We found a pretty reliable reproducer for this bug. Unfortunately it is only working on one of our classified filesystems, so I can't send debug logs. The server is running 2.4.0-24chaos (see <a href="https://github.com/chaos/lustre" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/chaos/lustre</a>) with ZFS and the clients are 2.4.0-19chaos. The reproducer is pretty simple. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># Create files locally then list them on remote node e8. for ((i=0;i&lt;20;i++)) ; do dd if=/dev/urandom of=file$i bs=1k count=1 ; done ; rsh e8 ls -l `pwd` </pre> </div></div> <p>Create several 1k files then immediately list them on another node. Some of the files are listed with 0 length, then show the correct lengths if listed again. Usually between 0 and 3 files are affected, but which and how many files varies between attempts.</p> <p>I captured debug logs from the client and MDS for one successful attempt. The client logs had <tt>+dlmtrace +rpctrace</tt> and the MDS log had -1. The bug wouldn't reproduce with -1 debugging on the clients. But, I haven't been able to find the bug yet from the logs. Please let me know if you have any tips on how to debug this. Meanwhile I'll keep trying to reproduce this on an unclassified system so we can send debug logs.</p> <p>I managed to reproduce the bug on an unclassified system and get debug logs from the clients, MDS, and OSS. I uploaded them in a tarball to ftp.whamcloud.com. Email me privately if you need the file name. There is a README in the tarball with a few notes of relevance.</p> <p>In case it helps interpret the debug logs, here are the NIDs of the nodes involved.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sierra654: 192.168.114.155@o2ib5 # created files sierra330: 192.168.113.81@o2ib5 # got zero length for file4 porter44: 172.19.1.213@o2ib100 # OSS owning object for file4 porter-mds1: 172.19.1.165@o2ib100 # MDS </pre> </div></div> <p>FWIW, having full debugging enabled on the servers seems to make this bug much easier to reproduce.</p> <p>I can reproduce it with two mounts, and looks it's a race of agl vs normal getattr (because it can't be reproduced anymore when agl turned off), will look into it further. </p> <p>I confirmed that disabling statahead_agl seem to prevent the bug here as well.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param llite.*.statahead_agl=0 </pre> </div></div> <p>patch for master: <a href="http://review.whamcloud.com/9249" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9249</a></p> <p>We set <tt>statahead_agl=0</tt> on all our clients to workaround this issue until the patch can be deployed. This seemed to work, however I just learned that the 'size changed from 0' error was reported for 9 files during a run of the "htar" archival storage utility. So there may be another (less frequent) bug that can cause this behavior.</p> <p>Patch for b2_5 at <a href="http://review.whamcloud.com/#/c/9328/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9328/</a></p> <p>Landed for 2.5.1 and 2.6</p> Development Rank 1|hzwehj: Rank (Obsolete) 12564 Severity