[LU-13194] Interop: sanityn test 104 fails with '/mnt/lustre/f104.sanityn mtime (1580286855:1580286853) diff' - Whamcloud Community JIRA

Details

Type: Bug
Resolution: Fixed
Priority: Minor
Fix Version/s: Lustre 2.14.0, Lustre 2.12.4
Affects Version/s: Lustre 2.10.8, Lustre 2.12.3
Labels:
- interop

Severity:
3
Rank (Obsolete):
9223372036854775807

Description

sanityn test_104 fails for clients with Lustre versions that are newer than the client version starting with Lustre 2.12.3 servers and 2.12.58.171 clients starting on 24 OCT 2019; see https://testing.whamcloud.com/test_sets/591b7932-f793-11e9-a197-52540065bddc and https://testing.whamcloud.com/test_sets/d58f7e54-f767-11e9-add9-52540065bddc.

Looking at a recent failure, https://testing.whamcloud.com/test_sets/8d9bdf58-4333-11ea-86c7-52540065bddc, for Lustre servers 2.11.0 and clients 2.12.3.109, we see the mtimes produced

CMD: trevis-51vm12 debugfs -c -R 'stat ROOT/f104.sanityn' /dev/mapper/mds1_flakey
trevis-51vm12: debugfs 1.45.2.wc1 (27-May-2019)
trevis-51vm12: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
STAT a|m|ctime 1580286853 1580286853 1580286853
MDT a|m|ctime 1580286853 1580286853 1580286853
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00156288 s, 655 kB/s
CMD: trevis-51vm12 debugfs -c -R 'stat ROOT/f104.sanityn' /dev/mapper/mds1_flakey
trevis-51vm12: debugfs 1.45.2.wc1 (27-May-2019)
trevis-51vm12: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
STAT a|m|ctime 1580286853 1580286855 1580286855
MDT a|m|ctime 1580286853 1580286855 1580286855
 sanityn test_104: @@@@@@ FAIL: /mnt/lustre/f104.sanityn mtime (1580286855:1580286853) diff

The patch for ~~LU-12026~~, https://review.whamcloud.com/36286/, added test 104 to sanityn. It landed to master on October 2019 and landed to b2_12 on 17 JAN 2020.

sanityn test 104 does not have a Lustre version check, but needs one. Master, future 2.14.0, interop testing runs with 2.12.3 and 2.13.0. 2.13.0 already has this patch and 2.12.4 has this patch. So, when we move interop testing for master to use 2.12.4, we don’t have to worry about master interop testing hitting this issue. The only other release we need to worry about is the LTS branch b2_12. The 2.12 branch runs interop testing with 2.11.0 and 2.10.8. Thus, if we land a patch to b2_12 that checks the server version to make sure it is 2.12.3.110 or later, then all b2_12 interop testing should skip this test when appropriate.

Attachments

Issue Links

mentioned in: Page No Confluence page found with the given URL.

Activity

[LU-13194] Interop: sanityn test 104 fails with '/mnt/lustre/f104.sanityn mtime (1580286855:1580286853) diff'

Peter Jones made changes - 08/Feb/20 5:50 AM

Fix Version/s

New: Lustre 2.12.4 [ 14690 ]

Gerrit Updater added a comment - 08/Feb/20 5:39 AM

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37420/
Subject: ~~LU-13194~~ tests: check server version sanityn 104
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: dca549213447dc3958ca225aae386a48c9757618

Gerrit Updater added a comment - 08/Feb/20 5:39 AM Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37420/ Subject: LU-13194 tests: check server version sanityn 104 Project: fs/lustre-release Branch: b2_12 Current Patch Set: Commit: dca549213447dc3958ca225aae386a48c9757618

Peter Jones made changes - 07/Feb/20 4:01 AM

Fix Version/s		New: Lustre 2.14.0 [ 14490 ]
Resolution		New: Fixed [ 1 ]
Status	Original: Open [ 1 ]	New: Resolved [ 5 ]

Peter Jones added a comment - 07/Feb/20 4:01 AM

Landed for 2.14

Peter Jones added a comment - 07/Feb/20 4:01 AM Landed for 2.14

Gerrit Updater added a comment - 06/Feb/20 7:50 PM

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37461/
Subject: ~~LU-13194~~ tests: check server version sanityn 104
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a8b9a123fea3762b999e80f56fdbbdf2ea10e280

Gerrit Updater added a comment - 06/Feb/20 7:50 PM Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37461/ Subject: LU-13194 tests: check server version sanityn 104 Project: fs/lustre-release Branch: master Current Patch Set: Commit: a8b9a123fea3762b999e80f56fdbbdf2ea10e280

Gerrit Updater added a comment - 06/Feb/20 3:13 PM

James Nunez (jnunez@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/37461
Subject: ~~LU-13194~~ tests: check server version sanityn 104
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6e64326566cc6963bc2896183fe84ec6b21d302a

Gerrit Updater added a comment - 06/Feb/20 3:13 PM James Nunez (jnunez@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/37461 Subject: LU-13194 tests: check server version sanityn 104 Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 6e64326566cc6963bc2896183fe84ec6b21d302a

James Nunez (Inactive) made changes - 05/Feb/20 12:09 AM

Remote Link

New: This issue links to "Page (Whamcloud Community Wiki)" [ 24480 ]

Andreas Dilger added a comment - 04/Feb/20 9:09 AM

What is a bit confusing is that it looks like the test should be passing:

STAT a|m|ctime 1580286853 1580286855 1580286855
MDT a|m|ctime 1580286853 1580286855 1580286855
 sanityn test_104: @@@@@@ FAIL: /mnt/lustre/f104.sanityn mtime (1580286855:1580286853) diff

The respective atimes, mtimes, and ctimes are the same between the client and MDT, even if they are not the same as each other. Either the STAT or MDT lines are being printed incorrectly, or the test is checking the wrong values?

Andreas Dilger added a comment - 04/Feb/20 9:09 AM What is a bit confusing is that it looks like the test should be passing: STAT a|m|ctime 1580286853 1580286855 1580286855 MDT a|m|ctime 1580286853 1580286855 1580286855 sanityn test_104: @@@@@@ FAIL: /mnt/lustre/f104.sanityn mtime (1580286855:1580286853) diff The respective atimes, mtimes, and ctimes are the same between the client and MDT, even if they are not the same as each other. Either the STAT or MDT lines are being printed incorrectly, or the test is checking the wrong values?

Gerrit Updater added a comment - 04/Feb/20 4:16 AM

James Nunez (jnunez@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/37420
Subject: ~~LU-13194~~ tests: check server version sanityn 104
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 67dbba5b73fc148596e81f801d3d92a9f5d38dbe

Gerrit Updater added a comment - 04/Feb/20 4:16 AM James Nunez (jnunez@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/37420 Subject: LU-13194 tests: check server version sanityn 104 Project: fs/lustre-release Branch: b2_12 Current Patch Set: 1 Commit: 67dbba5b73fc148596e81f801d3d92a9f5d38dbe

James Nunez (Inactive) made changes - 04/Feb/20 4:09 AM

Description

Original: sanityn test_104 fails for clients with Lustre versions that are newer than the client version starting with Lustre 2.12.3 servers and 2.12.58.171 clients starting on 24 OCT 2019; see https://testing.whamcloud.com/test_sets/591b7932-f793-11e9-a197-52540065bddc and https://testing.whamcloud.com/test_sets/d58f7e54-f767-11e9-add9-52540065bddc.

Looking at a recent failure, https://testing.whamcloud.com/test_sets/8d9bdf58-4333-11ea-86c7-52540065bddc, for Lustre servers 2.11.0 and clients 2.12.3.109, we see the mtimes produced
{noformat}
CMD: trevis-51vm12 debugfs -c -R 'stat ROOT/f104.sanityn' /dev/mapper/mds1_flakey
trevis-51vm12: debugfs 1.45.2.wc1 (27-May-2019)
trevis-51vm12: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
STAT a|m|ctime 1580286853 1580286853 1580286853
MDT a|m|ctime 1580286853 1580286853 1580286853
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00156288 s, 655 kB/s
CMD: trevis-51vm12 debugfs -c -R 'stat ROOT/f104.sanityn' /dev/mapper/mds1_flakey
trevis-51vm12: debugfs 1.45.2.wc1 (27-May-2019)
trevis-51vm12: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
STAT a|m|ctime 1580286853 1580286855 1580286855
MDT a|m|ctime 1580286853 1580286855 1580286855
sanityn test_104: @@@@@@ FAIL: /mnt/lustre/f104.sanityn mtime (1580286855:1580286853) diff
{noformat}

The patch for ~~LU-12026~~, https://review.whamcloud.com/#/c/36286/, added test 104 to sanityn. It landed to master on October 2019 and landed to b2_12 on 17 JAN 2020.

sanityn test 104 does not have a Lustre version check, but needs one. Master, future 2.14.0, interop testing runs with 2.12.3 and 2.13.0. 2.13.0 already has this patch and 2.12.4 has this patch. So, when we move interop testing for master to use 2.12.4, we don’t have to worry about master interop testing hitting this issue. The only other release we need to worry about is the LTS branch b2_12. The 2.12 branch runs interop testing with 2.11.0 and 2.10.8. Thus, if we land a patch to b2_12 that checks the server version to make sure it is 2.12.3.110 or later, then all b2_12 interop testing should skip this test when appropriate.

New: sanityn test_104 fails for clients with Lustre versions that are newer than the client version starting with Lustre 2.12.3 servers and 2.12.58.171 clients starting on 24 OCT 2019; see https://testing.whamcloud.com/test_sets/591b7932-f793-11e9-a197-52540065bddc and https://testing.whamcloud.com/test_sets/d58f7e54-f767-11e9-add9-52540065bddc.

Looking at a recent failure, https://testing.whamcloud.com/test_sets/8d9bdf58-4333-11ea-86c7-52540065bddc, for Lustre servers 2.11.0 and clients 2.12.3.109, we see the mtimes produced
{noformat}
CMD: trevis-51vm12 debugfs -c -R 'stat ROOT/f104.sanityn' /dev/mapper/mds1_flakey
trevis-51vm12: debugfs 1.45.2.wc1 (27-May-2019)
trevis-51vm12: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
STAT a|m|ctime 1580286853 1580286853 1580286853
MDT a|m|ctime 1580286853 1580286853 1580286853
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.00156288 s, 655 kB/s
CMD: trevis-51vm12 debugfs -c -R 'stat ROOT/f104.sanityn' /dev/mapper/mds1_flakey
trevis-51vm12: debugfs 1.45.2.wc1 (27-May-2019)
trevis-51vm12: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps
STAT a|m|ctime 1580286853 1580286855 1580286855
MDT a|m|ctime 1580286853 1580286855 1580286855
sanityn test_104: @@@@@@ FAIL: /mnt/lustre/f104.sanityn mtime (1580286855:1580286853) diff
{noformat}

The patch for ~~LU-12026~~, https://review.whamcloud.com/36286/, added test 104 to sanityn. It landed to master on October 2019 and landed to b2_12 on 17 JAN 2020.

sanityn test 104 does not have a Lustre version check, but needs one. Master, future 2.14.0, interop testing runs with 2.12.3 and 2.13.0. 2.13.0 already has this patch and 2.12.4 has this patch. So, when we move interop testing for master to use 2.12.4, we don’t have to worry about master interop testing hitting this issue. The only other release we need to worry about is the LTS branch b2_12. The 2.12 branch runs interop testing with 2.11.0 and 2.10.8. Thus, if we land a patch to b2_12 that checks the server version to make sure it is 2.12.3.110 or later, then all b2_12 interop testing should skip this test when appropriate.

People

Assignee:: James Nunez (Inactive)

Reporter:: James Nunez (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 03/Feb/20 11:42 PM

Updated:: 08/Feb/20 5:50 AM

Resolved:: 07/Feb/20 4:00 AM