Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Lustre 2.8.0, Lustre 2.9.0, Lustre 2.10.0, Lustre 2.11.0, Lustre 2.10.5
-
None
-
3
-
9223372036854775807
Description
This issue was created by maloo for Andreas Dilger <andreas.dilger@intel.com>
This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/9313c1e0-b06f-11e5-bf32-5254006e85c2.
The sub-test test_82a failed with the following error:
/usr/bin/lfs getstripe -c /mnt/lustre/d82a.conf-sanity/f82a.conf-sanity-1 wrong: found 2, expected 3
Looks like this might be related to running short of precreated OST objects on one of the OSTs and it is skipped rather than blocking the create. The MDS should allow at most 1/4 of requested stripes to be skipped if they have no objects rather than blocking the create indefinitely. However, it appears that this functionality was broken with the change from LOV to LOD, and in this case all 3 OST objects are required since (3 * 1/4 < 1) so no whole stripe could be skipped yet.
In lod_qos_prep_create() it does not set the flags = LOV_USES_DEFAULT_STRIPE for the cases when a filesystem-wide default striping is used as was done in the original qos_prep_create(), and as such lod_alloc_qos() requires that all requested stripes to be allocated. The lod_alloc_qos() code will fall back to lod_alloc_rr() with -EAGAIN if these cannot be allocated. In lod_alloc_rr() it will return success if at least one OST object was allocated, which doesn't seem correct if a large number of stripes was requested, though it isn't clear why lod_alloc_rr() doesn't wait for the OSTs to come online and allocate the requested number of objects.
Also, it looks like the check for lod_qos_is_usable() could be moved to the start of lod_alloc_qos() instead of after the pools are checked, since it doesn't use any of the pool information anyway.
Info required for matching: conf-sanity 82a
Attachments
Issue Links
- duplicates
-
-
- Resolved
-
- mentioned in
-
Page Loading...
-
-
-
-
-
-
-
-
-
-
Activity
This test would benefit from printing out the "$ost_indices" value and the actual file layout from "lfs getstripe" in case of error. Also, getting "lctl get_param osc..prealloc__id" on the MDS before and after the test is run would tell us if the OSTs have precreated objects, or if wait_osts_up is not enough. It would also be good to make it more clear which of the "wrong:" messages is being printed. I suspect that it is just a matter of waiting for the MDS-OSS connections to have preallocated objects.
hit again on master https://testing.hpdd.intel.com/test_sets/515a8c78-babf-11e7-9abd-52540065bddc
seen again on b2_10:
https://testing.hpdd.intel.com/test_sets/000cf930-9901-11e7-b778-5254006e85c2
I think the fix https://review.whamcloud.com/27441 is only landed on master, not b2_10.
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/27441/
Subject: LU-7631 tests: wait_osts_up waits for MDS precreates
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: edb0fb241bb5e0cc95c240ed977abf7f234ee045
Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: https://review.whamcloud.com/27441
Subject: LU-7631 tests: wait_osts_up waits for MDS precreates
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: cb7c2473cdbc2e375182e3d7de1b0fbfa6b0865a
I can't see why this test is formatting a new filesystem? It should be able to run with any existing filesystem, and this should also avoid the failure since it there will not be a startup issue with the OSTs not being ready.
I took a look at all the conf-sanity test 82a failures from the past almost five months, July 1 to November 19, and this test is only failing for 2.10.5 and 2.10.6 testing. I will upload a patch with Andreas suggestions to help with debugging in case we see this test fail again.