https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-221 Lustre <p>I hit this problem several times while running sanityn:</p> <p>== sanityn test 4: fstat validation on multiple mount points ========================================= 19:19:59 (1303179599)<br/> Mtimes don't match 1303179601, 1303179727<br/> sanityn test_4: @@@@@@ FAIL: test_4 failed with 1 </p> <p>test result on maloo:<br/> <a href="https://maloo.whamcloud.com/test_sets/4d51c720-6a37-11e0-b32b-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4d51c720-6a37-11e0-b32b-52540025f9af</a></p> <p>It's same to bug 18169.</p> 2.6.18-194.17.1.el5 LU-221

sanityN.sh test_4 failed

Bug Blocker Resolved Fixed Hongchao Zhang Niu Yawei Mon, 18 Apr 2011 21:08:18 +0000 Wed, 3 Dec 2014 23:28:20 +0000 Wed, 27 Jul 2011 08:18:47 +0000 Lustre 2.1.0 Lustre 1.8.6 Lustre 2.1.0 0 3 <p>HongChao</p> <p>Could you please look into this failure?</p> <p>Thanks</p> <p>Peter</p> <p>Okay, take this issue.</p> <p>this issue should be caused by the time difference between client and OST,<br/> the atime of the file "f4" is even larger than the current time in client,</p> <p>00000020:00010000:3.0:1303179601.011757:0:21145:0:(cl_object.c:308:cl_object_glimpse()) size: 5 mtime: 1303179601 atime: 1303179727 ctime: 1303179601 blocks: 0</p> <p>the mtime and ctime is sane, but atime(1303179727) is "126" seconds later than the current time(1303179601.011757)</p> <p>in filter, the LVB will be initialized according to the value of its object, then<br/> for client1, the ctime will be the current time at client for it just write some data and there is no need to send glimpse lock request,<br/> but for client2, the glimpse lock will be sent to update the LVB and the ctime will be set to the time in OST (1303179727)</p> <p>Hi, Hongchao</p> <p>The test_4 is to compare mtime on two clients, but not ctime or atime. I think in theory, the mtime seen by client2 should be the mtime on client1, and I didn't see why server time is involved, did I miss anything?</p> <p>After look closer to the code, I see that the time difference between OSS and client could affect the test result:</p> <ul class="alternate" type="square"> <li>client 1 create file on time_1, MDS create file and set mtime as client1's time_1, the mtime of object on OSS is OSS local server time time_s; (time_s &gt; time_1, if OSS clock is faster than the client_1)</li> </ul> <ul class="alternate" type="square"> <li>client 2 open file, it get time_1 from MDS, and glimpse time_s from OSS, because time_s &gt; time_1, it take the time_s as it's mtime;</li> </ul> <ul class="alternate" type="square"> <li>client 1 write to the file on client1's time_2;</li> </ul> <ul class="alternate" type="square"> <li>client 1 stat file, it take local time_2 as mtime; (since time_s &gt; time_2, no matter if the dirty is flushed, the mtime of object on OSS should always be time_s)</li> </ul> <ul class="alternate" type="square"> <li>client 2 stat file, it glimpse time from OSS, OSS issues glimpse callback to client1, but because the time_s is even later than client1's time_2, client2 still take the later one time_s as it's mtime;</li> </ul> <ul class="alternate" type="square"> <li>at last, client1's mtime (time_2) != client2's mtime (time_s);</li> </ul> <p>Hi, Hongchao</p> <p>Is this what you described in your previous comment? I'm wondering if it's a bug, in my opinion, lustre should always take client's time as file's a/c/mtime, server time should never been involved, but if we do require the time be synced on all nodes (include clients and servers), then it's not neccesarry a bug.</p> <p>If we regard it as a bug, I think the fix could be quite simple, we can just set the object's a/c/mtime as zero (or a very small time) when pre-creating them, then the untouched objects on OSS will no longer contribute time value in the glimpse mechanism, until it's written or truncated by some client.</p> <p>Hi, Andreas, any comments on this?</p> <p>It should already be possible to handle this today by checking the SUID/SGID flags on the inode, and if both of them are set then the object a/m/ctime should be ignored (return 0 for all of them). I'd prefer not to set those values to 0 on disk because some parts of the code consider ctime = 0 an invalid inode. </p> <p>It is definitely correct that the times should be based on the client and not the server. While it is desirable to have clock in sync on all nodes, this is not required. </p> <p>Any progress on this bug?</p> <p>I see this is still failing in Maloo, e.g. <a href="https://maloo.whamcloud.com/test_sets/fa0a9a24-a78f-11e0-bd2a-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fa0a9a24-a78f-11e0-bd2a-52540025f9af</a>. Fixing the failing regression tests makes our testing much more efficient.</p> <p>I think the proposed fix is relatively straight forward to implement - in filter_commitrw_write(), if <em>both</em> SUID and SGID are still set (i.e. this is the first time this was done), then OBD_MD_FLMTIME|OBD_MD_FLCTIME|OBD_MD_FLATIME should be OR'd into the "i" flag passed to iattr_from_obdo(), and ATTR_MTIME | ATTR_CTIME | ATTR_ATIME should be left in the ia_valid mask, so that the values sent from the client overwrite those on disk in the call to fsfilt_setattr().</p> <p>the initial patch has been created (some improvement is still needed), but it was suspended for there are several other high<br/> priority bugs needed to investigate. I'll complete it soon.</p> <p>I think that we need to make fixing this issue a priority. As Andreas says it causes regular autotest failures</p> <p>the patch is at <a href="http://review.whamcloud.com/#change,1084" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1084</a></p> <p>some notes about the patch,<br/> 1, S_ISUID, S_ISGID can't be used to determine whether the inode's a/c/m time is valid, for the a/c/m can be set individually<br/> by users(say, "touch"), test_39m in sanity.sh just tests this case. </p> <p> in this patch, the a/c/m time is set as the minimal value of inode-&gt;i_a(m,c)time (LONG_MIN), and it will be ignored <br/> by clients for only the newest time can be used.</p> <p>2, in this patch, the a/c/m time is initialized after creating the inode, and it can be moved into ldiskfs if it degrade the<br/> creation performance.</p> <p>there is an error in the Maloo's test for this patch<br/> <a href="https://maloo.whamcloud.com/test_sets/b3babb64-b1fb-11e0-b33f-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b3babb64-b1fb-11e0-b33f-52540025f9af</a></p> <p>there is a bug (<a href="https://bugzilla.lustre.org/show_bug.cgi?id=23161" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=23161</a>) in buzilla that tracked this issue.</p> <p>not sure whether the new occurrence is related to the patch yet. </p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/231/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #231</a><br/> <a href="https://jira.whamcloud.com/browse/LU-221" title="sanityN.sh test_4 failed" class="issue-link" data-issue-key="LU-221"><del>LU-221</del></a> don't use a/c/m time for newly allocated object in OST</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=414251797ed178eec5d431e1f5aa4a889d2b159f" class="external-link" target="_blank" rel="nofollow noopener">414251797ed178eec5d431e1f5aa4a889d2b159f</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/231/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #231</a><br/> <a href="https://jira.whamcloud.com/browse/LU-221" title="sanityN.sh test_4 failed" class="issue-link" data-issue-key="LU-221"><del>LU-221</del></a> don't use a/c/m time for newly allocated object in OST</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=414251797ed178eec5d431e1f5aa4a889d2b159f" class="external-link" target="_blank" rel="nofollow noopener">414251797ed178eec5d431e1f5aa4a889d2b159f</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/231/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #231</a><br/> <a href="https://jira.whamcloud.com/browse/LU-221" title="sanityN.sh test_4 failed" class="issue-link" data-issue-key="LU-221"><del>LU-221</del></a> don't use a/c/m time for newly allocated object in OST</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=414251797ed178eec5d431e1f5aa4a889d2b159f" class="external-link" target="_blank" rel="nofollow noopener">414251797ed178eec5d431e1f5aa4a889d2b159f</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/231/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #231</a><br/> <a href="https://jira.whamcloud.com/browse/LU-221" title="sanityN.sh test_4 failed" class="issue-link" data-issue-key="LU-221"><del>LU-221</del></a> don't use a/c/m time for newly allocated object in OST</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=414251797ed178eec5d431e1f5aa4a889d2b159f" class="external-link" target="_blank" rel="nofollow noopener">414251797ed178eec5d431e1f5aa4a889d2b159f</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/231/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #231</a><br/> <a href="https://jira.whamcloud.com/browse/LU-221" title="sanityN.sh test_4 failed" class="issue-link" data-issue-key="LU-221"><del>LU-221</del></a> don't use a/c/m time for newly allocated object in OST</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=414251797ed178eec5d431e1f5aa4a889d2b159f" class="external-link" target="_blank" rel="nofollow noopener">414251797ed178eec5d431e1f5aa4a889d2b159f</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/231/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #231</a><br/> <a href="https://jira.whamcloud.com/browse/LU-221" title="sanityN.sh test_4 failed" class="issue-link" data-issue-key="LU-221"><del>LU-221</del></a> don't use a/c/m time for newly allocated object in OST</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=414251797ed178eec5d431e1f5aa4a889d2b159f" class="external-link" target="_blank" rel="nofollow noopener">414251797ed178eec5d431e1f5aa4a889d2b159f</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/231/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #231</a><br/> <a href="https://jira.whamcloud.com/browse/LU-221" title="sanityN.sh test_4 failed" class="issue-link" data-issue-key="LU-221"><del>LU-221</del></a> don't use a/c/m time for newly allocated object in OST</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=414251797ed178eec5d431e1f5aa4a889d2b159f" class="external-link" target="_blank" rel="nofollow noopener">414251797ed178eec5d431e1f5aa4a889d2b159f</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/231/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #231</a><br/> <a href="https://jira.whamcloud.com/browse/LU-221" title="sanityN.sh test_4 failed" class="issue-link" data-issue-key="LU-221"><del>LU-221</del></a> don't use a/c/m time for newly allocated object in OST</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=414251797ed178eec5d431e1f5aa4a889d2b159f" class="external-link" target="_blank" rel="nofollow noopener">414251797ed178eec5d431e1f5aa4a889d2b159f</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/231/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #231</a><br/> <a href="https://jira.whamcloud.com/browse/LU-221" title="sanityN.sh test_4 failed" class="issue-link" data-issue-key="LU-221"><del>LU-221</del></a> don't use a/c/m time for newly allocated object in OST</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=414251797ed178eec5d431e1f5aa4a889d2b159f" class="external-link" target="_blank" rel="nofollow noopener">414251797ed178eec5d431e1f5aa4a889d2b159f</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/231/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #231</a><br/> <a href="https://jira.whamcloud.com/browse/LU-221" title="sanityN.sh test_4 failed" class="issue-link" data-issue-key="LU-221"><del>LU-221</del></a> don't use a/c/m time for newly allocated object in OST</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=414251797ed178eec5d431e1f5aa4a889d2b159f" class="external-link" target="_blank" rel="nofollow noopener">414251797ed178eec5d431e1f5aa4a889d2b159f</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/231/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #231</a><br/> <a href="https://jira.whamcloud.com/browse/LU-221" title="sanityN.sh test_4 failed" class="issue-link" data-issue-key="LU-221"><del>LU-221</del></a> don't use a/c/m time for newly allocated object in OST</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=414251797ed178eec5d431e1f5aa4a889d2b159f" class="external-link" target="_blank" rel="nofollow noopener">414251797ed178eec5d431e1f5aa4a889d2b159f</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/231/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #231</a><br/> <a href="https://jira.whamcloud.com/browse/LU-221" title="sanityN.sh test_4 failed" class="issue-link" data-issue-key="LU-221"><del>LU-221</del></a> don't use a/c/m time for newly allocated object in OST</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=414251797ed178eec5d431e1f5aa4a889d2b159f" class="external-link" target="_blank" rel="nofollow noopener">414251797ed178eec5d431e1f5aa4a889d2b159f</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/231/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #231</a><br/> <a href="https://jira.whamcloud.com/browse/LU-221" title="sanityN.sh test_4 failed" class="issue-link" data-issue-key="LU-221"><del>LU-221</del></a> don't use a/c/m time for newly allocated object in OST</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=414251797ed178eec5d431e1f5aa4a889d2b159f" class="external-link" target="_blank" rel="nofollow noopener">414251797ed178eec5d431e1f5aa4a889d2b159f</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/231/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #231</a><br/> <a href="https://jira.whamcloud.com/browse/LU-221" title="sanityN.sh test_4 failed" class="issue-link" data-issue-key="LU-221"><del>LU-221</del></a> don't use a/c/m time for newly allocated object in OST</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&amp;a=commit&amp;h=414251797ed178eec5d431e1f5aa4a889d2b159f" class="external-link" target="_blank" rel="nofollow noopener">414251797ed178eec5d431e1f5aa4a889d2b159f</a><br/> Files : </p> <ul> <li>lustre/obdfilter/filter.c</li> </ul> <p>Landed for 2.1</p> Duplicate LU-222 Related LU-1042 LU-5977 Bugzilla ID 18169.0 Development Epic metadata Rank 1|hzv3hz: Rank (Obsolete) 4066 Severity