[LU-12058] sanity test_51d: @@@@@@ FAIL: OST 1 has less #0 objects vs OST 0 (0 < 1000

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.15.0
• Type: Bug
• Priority: Minor
• Reporter: Vladimir Saveliev
• Assignee: Vladimir Saveliev
• Resolution: Fixed
• Severity: 3

Description

sanity.sh:test_51d assumes that stripe offset is -1, so it fails otherwise.

#! /bin/bash
 
export OSTCOUNT=3
 
REFORMAT=yes sh llmount.sh 
lfs setstripe -i 0 -c 2 /mnt/lustre
NOFORMAT=yes ONLY=51d sh sanity.sh

The test's output:

== sanity test 51d: check object distribution ======================================================== 00:10:37 (1552252237)
total: 1000 open/close in 1.82 seconds: 549.05 ops/second
OST0 has 1000 objects, 1000 are index 0
OST1 has 1000 objects, 0 are index 0
OST2 has 0 objects, 0 are index 0
 - unlinked 0 (time 1552252239 ; total 0 ; last 0)
total: 1000 unlinks in 3 seconds: 333.333344 unlinks/second
 sanity test_51d: @@@@@@ FAIL: OST 1 has less #0 objects vs OST 0  (0 < 1000 

 

Issue Links

• is duplicated by: LU-14745 sanity: sanity test_51d: @@@@@@ FAIL...
• is related to: LU-8933 sanity test_51d: object distribution ...
• is related to: LU-9506 sanity test_51d: OST 3 has less objec...

Comments

[Gerrit Updater]

Vladimir Saveliev ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34390
Subject: LU-12058 tests: skip sanity 51d if stripe offset is unsuited
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 13ef4975074bfecd55f904b8b02395e06737e142

[Andreas Dilger]

Isn't this fallout from the previous patch https://review.whamcloud.com/34386 "LU-12055 lod: ignore root layout if parent's one includes pool" leaving the default "-i 0" striping on the whole filesystem? I'm not against fixing the test to not have assumptions about the test environment, but the other test should also be fixed not to change the global filesystem defaults in the middle of the test run.

[Alex Zhuravlev]

https://testing.whamcloud.com/test_sessions/6ff51064-1eac-40b4-997f-bc6fcf59607a

[Andreas Dilger]

+1 on master https://testing.whamcloud.com/test_sets/220ac885-cdcd-42f9-bd09-3c1160c27945

The original commit adding test_51d was:

Branch b1_4
Description: Inefficient object allocation for multi-stripe files
Details    : When selecting which OSTs to stripe files over, for files with
             a stripe count that divides evenly into the number of OSTs,
             the MDS is always picking the same starting OST for each file.
             Return the OST selection heuristic to the original design.

             Also make these variables per-LOV for the case of multiple
             MDTs on the same node and add sanity.sh test 51d to verify.
b=10671

I think a core problem with the test is that if the OSTs are imbalanced and QOS is used, then the starting OST allocation will also be imbalanced, causing the test to intermittently fail. Either it should explicitly disable QOS before the start of the test (*.*.qos_threshold=100, account for the OST imbalance, or just be more forgiving in the thresholds.

[Gerrit Updater]

"Andreas Dilger <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/44762
Subject: LU-12058 tests: improve sanity test_51d reliability
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0844979ef274be7269ab2be98db7815c352ebe1f

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/44762/
Subject: LU-12058 tests: improve sanity test_51d reliability
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 2e832a0038af7f3d9f022beb862578988592d682

[Peter Jones]

Landed for 2.15