[LU-17675] sanity-flr test_61a el9.3: FAIL: atime: old '1711375406' != new '1711375401'

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.16.0, Lustre 2.15.5
• Affects Version/s: Lustre 2.16.0
• Severity: 3
• Assignee: Alex Zhuravlev
• Reporter: Maloo

Description

This issue was created by maloo for jianyu <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/e2146721-7935-48e5-8cbc-7ab81a3be223

test_61a failed with the following error:

/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr: atime: old '1711375406' != new '1711375401'

Test session details:
clients: https://build.whamcloud.com/job/lustre-master/4514 - 5.14.0-284.30.1.el9_2.x86_64
servers: https://build.whamcloud.com/job/lustre-master/4514 - 5.14.0-284.30.1_lustre.el9.x86_64

<<Please provide additional information about the failure here>>

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity-flr test_61a - /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr: atime: old '1711375406' != new '1711375401'

Issue Links

• is related to: LU-16421 sanity-flr test_61a: atime: old '1670819992' != new '1670819987'
• is related to: LU-17265 sanity test_39r: atime on client 1699192823 != ost 0x65479ff6

Comments

[Alex Zhuravlev]

original timestamps
== timestamps for /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr
lma: fid=[0x200000404:0x4:0x0] compat=0 incompat=0
atime: 0x66198920:00000000 – Fri Apr 12 19:18:56 2024

• 0: { l_ost_idx: 3, l_fid: [0x3c0000403:0x2:0x0] }

  == O/3c0000403/d2/2 ==
  lma: fid=[0x3c0000403:0x2:0x0] compat=8 incompat=0
  atime: 0x66198920:00000000 – Fri Apr 12 19:18:56 2024

mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps
== timestamps for /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr
lma: fid=[0x200000404:0x4:0x0] compat=0 incompat=0
atime: 0x66198920:00000000 – Fri Apr 12 19:18:56 2024

• 0: { l_ost_idx: 3, l_fid: [0x3c0000403:0x2:0x0] }
• 0: { l_ost_idx: 4, l_fid: [0x400000401:0x2:0x0] }

  == O/3c0000403/d2/2 ==
  lma: fid=[0x3c0000403:0x2:0x0] compat=8 incompat=0
  atime: 0x66198920:00000000 – Fri Apr 12 19:18:56 2024
  == O/400000401/d2/2 ==
  lma: fid=[0x400000401:0x2:0x0] compat=8 incompat=0
  atime: 0x66198920:00000000 – Fri Apr 12 19:18:56 2024

sanity-flr test_61a: @@@@@@ FAIL: /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr: atime: old '1712949543' != new '1712949536'

(gdb) p/x 1712949543
$1 = 0x66198927
(gdb) p/x 1712949536
$2 = 0x66198920

i.e. what is claimed "old" timestamp is not presented neither on MDS nor on OSTs

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/54731/
Subject: LU-17675 tests: disable sanity-flr/61a on RHEL9.3
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: df0b73906e34e124ab94ef3ab7438f8d400d99c9

[Gerrit Updater]

"Alex Zhuravlev <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/54733
Subject: LU-17675 tests: collect debug info
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 842c1006fa17a0be19f66f839b45bb632b2648d9

[Gerrit Updater]

"Alex Zhuravlev <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/54731
Subject: LU-17675 tests: disable sanity-flr/61a on RHEL9.3
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: dce0441c1ff65ce80e90218ec21f35bd79041d4d

[Alex Zhuravlev]

seems to be a bit more complicated than expected:

Error: '/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr: atime: old '1712819228' != new '1712819221''

[Alex Zhuravlev]

Doh! Can you try adding a longer sleep into the test, so that atime is updated?

sure

[Gerrit Updater]

"Alex Zhuravlev <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/54729
Subject: LU-17675 tests: sleep longer in sanity-flr/61a
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ac72569aa787228d89695ae104f7517ecf931d5a

[Andreas Dilger]

Doh! Can you try adding a longer sleep into the test, so that atime is updated?

[Alex Zhuravlev]

Test-Parameters: trivial testlist=sanity env=ONLY=61a,ONLY_REPEAT=100 clientdistro=el9.3

just noticed the testing patch asked for sanity, not sanity-flr

[Alex Zhuravlev]

Error: '/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr: atime: old '1712732591' != new '1712732585''
Failure Rate: 32.61% of most recent 92 runs, 8 skipped (all branches)

seems to be a very popular issue

[Alex Zhuravlev]

still happens: https://testing.whamcloud.com/test_sets/16db3ccc-8594-4cea-ab6b-72b3fda2116d