https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1619 Lustre <p>The results of a Maloo automated test of the nanosecond timestamp patches (<a href="https://jira.whamcloud.com/browse/LU-1158" title="nanosecond timestamp support for Lustre" class="issue-link" data-issue-key="LU-1158">LU-1158</a>, <a href="http://review.whamcloud.com/#change,3266" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3266</a> and <a href="http://review.whamcloud.com/#change,3271" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,3271</a>, the latter of which includes a revised sanityn.sh for testing nanosecond times) revealed a slight time desynchronization after file renaming in test 39a of the revised sanityn.sh, on the order of 500,000 nanoseconds (about 1/2 of a millisecond) or so. This failure occurred 1 time out of 100 trials. Since the previous maximum time granularity of Lustre was at second resolution, and this bug cropped up 1% of the time during testing, it is highly unlikely that this bug would be noticed or reproducible without application of the <a href="https://jira.whamcloud.com/browse/LU-1158" title="nanosecond timestamp support for Lustre" class="issue-link" data-issue-key="LU-1158">LU-1158</a> patches; for it to be noticed, of the 1% of times the bug occurs, the 1/2 millisecond difference would have to occur across a second boundary, i.e. the second timestamps would have to be different. However, since the patches do not change any logic outside of determining when a timestamp is greater than another, it is possible that this bug exists undetected in master.</p> <p>Links to related Maloo test session and sanityn.sh failure:<br/> <a href="https://maloo.whamcloud.com/test_sessions/fd767d96-ca51-11e1-a9ff-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/fd767d96-ca51-11e1-a9ff-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/34a6a812-ca53-11e1-a9ff-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/34a6a812-ca53-11e1-a9ff-52540035b04c</a></p> CentOS 6.2/x86_64 on Maloo LU-1619

Subtle and infrequent failure of test 39a in sanityn.sh noticed after ns timestamp implementation

Technical task LU-1158 Minor Resolved Duplicate WC Triage Andreas Dilger ReviewNeeded Tue, 10 Jul 2012 17:03:53 +0000 Fri, 23 Jun 2017 18:41:42 +0000 Fri, 23 Jun 2017 18:41:42 +0000 Lustre 2.3.0 0 0 <p>comment<br/> Isami, thanks for filing this bug. It is always useful to track issues like this, even if they do not appear to be causing any significant problems, since I suspect it is a sign that the timestamps aren't being handled correctly somewhere, and could lead to much larger skews in the timestamps. just a note about the "failure rate" of test results in Maloo. The percentage reported in Maloo means "1 of the past 100 tests failed" (i.e. the test just run and failed), so it isn't at all clear whether this means "every test from now on with this patch will fail" or "this was a fluke and the failure rate is 1 in 10000". If you want to get statistically significant numbers on how often this is failing, please try something like: </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> Test-Parameters: testlist=sanityn,sanityn,sanityn,sanityn,sanityn,sanityn,sanityn envdefinitions=ONLY=39 </pre> </div></div> <p> or similar, so that it runs sanityn test_39* several times. There typically shouldn't be any "drift" in the timestamps in this manner - everything should be derived only from the client clock. There are a couple of places that this might be introduced: - different nanosecond times on different CPU cores due to clock drift. I doubt this is the case anymore, but this used to happen in the past - skew between userspace and kernel timestamps. This could only be the case if the test is somehow using the clock from userspace - bug in the code allowing the MDS or OSS timestamps to affect the file. This is the most likely cause. It would be possible to test out the last hypothesis by setting wildly different times on the MDS and OSS nodes from the client (either or both in the past or future), and then tracing where the bad timestamp is coming from (which should be easy if the timestamps can clearly be distinguished as to which node they came from).</p> Duplicate LU-6366 Development Rank 1|hzw1t3: Rank (Obsolete) 10374 Story Points 3.0