[LU-17265] sanity test_39r: atime on client 1699192823 != ost 0x65479ff6

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0
• Severity: 3
• Assignee: Arshad Hussain
• Reporter: Maloo

Description

This issue was created by maloo for Arshad <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/873c927b-d963-40fa-830f-f3caea45a955

test_39r failed with the following error:

atime on client 1699192823 != ost 0x65479ff6

Test session details:
clients: https://build.whamcloud.com/job/lustre-reviews/100162 - 4.18.0-477.21.1.el8_8.aarch64
servers: https://build.whamcloud.com/job/lustre-reviews/100162 - 4.18.0-477.21.1.el8_lustre.x86_64

<<Please provide additional information about the failure here>>

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity test_39r - atime on client 1699192823 != ost 0x65479ff6

Issue Links

• duplicates: LU-16421 sanity-flr test_61a: atime: old '1670819992' != new '1670819987'
• is related to: LU-17675 sanity-flr test_61a el9.3: FAIL: atime: old '1711375406' != new '1711375401'
• is related to: LU-18356 test-framework to fetch except list from server for interop testing
• is related to: LU-13578 sanity test_39r: atime on client != ost
• is related to: LU-14091 sanity test_39r: 'atime on client 1603945630 != ost 0x5f9a449d'

Comments

[Peter Jones]

Landed for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/53035/
Subject: LU-17265 tests: allow margin for sanity/39r
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c5aa16db172afc9cbf0d4fd2c85261fef1a40d7b

[Gerrit Updater]

"Arshad Hussain <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53035
Subject: LU-17265 tests: Add 2s (leniency) in comparing timestamps
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a166c6aed4dc3b1114969d1b3eb8ce5ce26a4370

[Arshad Hussain]

Sure. I have assigned it to my name and thanks for explaination on the problem.

[Andreas Dilger]

Arshad, any chance you can push a fix for this, it should just be a one-line test script fix.

[Andreas Dilger]

The comparisons are done within "$((...))" so there is hex-to-decimal conversion, but it is a bit confusing in the error message.

Lustre should never be using the server timestamps when storing attributes on the servers. That avoids problems like NFS where the client gets tons of errors during a build when the client and server have different timestamps.

That said, it seems possible that there is a race between when the client is sending the read RPCs for "dd" and when the client VFS updates the attributes on the local inode. We do not send another RPC to the OST to update the timestamps, so it might be a second outdated, but shouldn't be 10s outdated (ie. didn't get updated at all).

So it seems the test should be lenient for a 1s (or 2s?) difference in the timestamps, and that will still achieve the goals of the test.

[Arshad Hussain]

 

atime on client 1699192823 != ost 0x65479ff6

Two obervations here:

 

• Compression is done between decimal vs hex. Which is not wrong. Maybe bringing it to same unit would not be wrong

• The drift is just  1 second. (Time on server is 1 second early than what was read on client)