[LU-16541] sanity test_64f: buffered io, not write rpc: grants mismatch: 12656640, expected 4218880

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.16.0, Lustre 2.15.4
• Affects Version/s: Lustre 2.16.0
• Labels: arm
• Severity: 3
• Assignee: Patrick Farrell
• Reporter: Maloo

Description

This issue was created by maloo for Arshad <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/a0c04b66-87b8-4b06-aee8-8ae97f9e229d

test_64f failed with the following error:

buffered io, not write rpc: grants mismatch: 12656640, expected 4218880

Test session details:
clients: https://build.whamcloud.com/job/lustre-reviews/92173 - 4.18.0-372.32.1.el8_6.aarch64
servers: https://build.whamcloud.com/job/lustre-reviews/92173 - 4.18.0-348.23.1.el8_lustre.x86_64

<<Please provide additional information about the failure here>>

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity test_64f - buffered io, not write rpc: grants mismatch: 12656640, expected 4218880

Issue Links

• is related to: LU-16673 sanity test_125: failures with aarch64 servers

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52096/
Subject: LU-16541 tests: Improve test 64f
Project: fs/lustre-release
Branch: b2_15
Current Patch Set:
Commit: 478ca310b4204b4354245e6261edae6fef0ae497

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52096
Subject: LU-16541 tests: Improve test 64f
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: 52d69926bb180b4d8e7ebf30f5d27431da544c17

[Peter Jones]

Merged for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52040/
Subject: LU-16541 tests: Improve test 64f
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 33e4d86a480b860e0a3b4b51c7c6da6ec0159e51

[Gerrit Updater]

"Patrick Farrell <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52040
Subject: LU-16541 tests: Improve test 64f
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 203bdc67a292bb1daff50d379d7dbfe5a86cc89c

[Oleg Drokin]

he other annoyance with this test failure btw is if it hits, it takes 1 hour for the test to finish which makes it a timeout on janitor:

sanity.test_64f.test_log.oleg346-client.log2023-08-19 04:46  1.6K 
sanity.test_64g.test_log.oleg346-client.log2023-08-19 05:46  1.6K 

Example here, but they ar eall like this. I wonder if the unbounded wait just waits for something else that happens to run for an hour? Nothing obvious in the test output:

http://testing.linuxhacker.ru/lustre-reports/33659/testresults/sanity2-ldiskfs-DNE-centos7_x86_64-centos7_x86_64/

the one hour duration also holds for the maloo

[Gerrit Updater]

"Patrick Farrell <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52023
Subject: LU-16541 tests: bisect
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a042bf7d62ec99b8b6cc299be998408286dd2b2a

[Gerrit Updater]

"Patrick Farrell <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/52022
Subject: LU-16541 tests: bisect
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9a360210bff4e076ae59171fe32c3656184a5901

[Patrick Farrell]

Ah, thank you.

[Andreas Dilger]

It looks like the spike in subtest failures on master started on 2023-08-02.