[LU-18676] sanity-quota test_49: block usage for user ID 10 is wrong 0, 2

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.17.0
• Affects Version/s: Lustre 2.16.1
• Labels: janitor9x zfs-2.2.7 zfs-2.3
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Maloo

Description

This issue was created by maloo for Andreas Dilger <[e-mail]>

This issue relates to the following test suite run:
https://testing.whamcloud.com/test_sets/8f73fd31-5306-453d-b487-920bca8adb5b
https://testing.whamcloud.com/test_sets/218e8071-62c6-4592-8436-2ad404d43736

test_49 failed with the following error:

block usage for user ID 10 is wrong 0, 2

Test session details:
clients: https://build.whamcloud.com/job/lustre-reviews/109554 - 5.14.0-503.15.1.el9_5.x86_64
servers: https://build.whamcloud.com/job/lustre-reviews/109554 - 5.14.0-503.15.1_lustre.el9.x86_64

Have seen this failure intermittently since 2024-12-10, but has started failing a lot since 2025-01-21 on master (10x or more per day) during review testing.

The test output doesn't show the UID so it is hard to see what is going on with the test. It might be that the file writes are taking longer than expected to commit to storage so the block count is not updated? However, none of the other IDs are reporting an error. It might be that they are all wrong, and this is just the first one checked and then the test exits?

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity-quota test_49 - block usage for user ID 10 is wrong 0, 2

Issue Links

• is related to: LU-18646 ost-pools test_23b: FAIL: dd didn't fail with ENOSPC (26214400 > 22639616)
• is related to: LU-18387 RHEL 9.5 support

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/57997/
Subject: LU-18676 tests: enable sanity-quota_49
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: ce20d5a6cb529bc6ee2f51e5c7db761fdff5f21e

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/58083/
Subject: LU-18676 tests: random write to set file size
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 149657180e53bc1cfee876bb4ced72853a10b09c

[Gerrit Updater]

"Hongchao Zhang <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/58083
Subject: LU-18676 osd-zfs: write to create file size
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 70e3587d9918ca14aaf4fdd5798b8f97debeec5d

[Sergey Cheremencev]

"LU-16780 osd: check all commit callbacks done" doesn't help. It still fails.

[Gerrit Updater]

"Sergey Cheremencev <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/58005
Subject: LU-18676 osd: check all commit callbacks done
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 3bcaa3119ac51228380f96a04b713d69fb55ab15

[Sergey Cheremencev]

The test output doesn't show the UID so it is hard to see what is going on with the test. It might be that the file writes are taking longer than expected to commit to storage so the block count is not updated?

It seems so.

I've pushed https://review.whamcloud.com/c/fs/lustre-release/+/57997 to ensure it is still easy reproducible. Plus it would print the first 50 lines instead of the tail. There is a chance that https://review.whamcloud.com/c/fs/lustre-release/+/50790 might help with the problem. I am going to rebase 50790 on 57997 if 57997 would reproduce the problem.

[Gerrit Updater]

"Sergey Cheremencev <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/57997
Subject: LU-18676 tests: enable sanity-quota_49
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 727d4145e21bec66e4ad6b8bc6bfdc4e3abe2529