[LU-10808] DoM: component end should align with dom_stripesize

• Type: Technical task
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.12.0
• Affects Version/s: Lustre 2.11.0
• Labels: DoM2
• Assignee: Mikhail Pershin
• Reporter: Andreas Dilger

Description

If the DoM component extent_end is set larger than the MDT dom_stripesize (via lfs setstripe -Eextent_end -L mdt ) then this currently generates an error.  Since users do not have any easy way to determine the dom_stripesize on the MDT, and it may in fact be different on a per-MDT basis (e.g. if MDT0000 sets dom_stripesize=0 because it was formatted before DoM, or it is adjusted automatically by the MDS when the MDT is nearly full).  This complicates DoM usage for users (imagine a striped directory that has different DoM size limits for the MDTs it is striped over).

The MDS should automatically adjust the component extent_end to match the MDT dom_stripesize, and if dom_stripesize=0 then the DoM component should be removed.

Issue Links

• is related to: LU-10786 sanity-flr test_45: Create /mnt/lustre/d45.sanity-flr/f45.sanity-flr failed
• is related to: LU-10917 If setstripe for DoM file fails, less/no information is given
• is related to: LU-11608 DoM2: inherited DoM component size is not limited
• is related to: LU-10070 PFL self-extending file layout
• is related to: LU-10465 increase default stripe size to 4MB

Comments

[Peter Jones]

Seems like all patches have landed for 2.12

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32482/
Subject: LU-10808 lod: remove DoM component if DoM is disabled
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3fd758ed4f40f235d26df9fd7c6b459590fbe0cf

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32073/
Subject: LU-10808 lod: align wrong DoM stripe values with defaults
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9146d261f35b394e10afde3eec2d5895425261e0

[Andreas Dilger]

I'm not suggesting to disable DoM permanently if one MDT is full, only on a per-file basis. Firstly, MDT is the one that selects the file layout and should decide if it is full, and this can be done on a per-MDT basis.

Secondly, the hash of the filename is what decides which MDT is used, so we can't try different MDTs to find space for the DoM data.

[Patrick Farrell (Inactive)]

This question has likely been answered elsewhere, but yes, it's possible to know if a layout is a default layout.  At file creation time, there's a moment where you inherit from a template.  But after that there's no way to know, currently.

It seems like your suggestion is if any MDT in the set is low on space, then remove the DOM component from the layout (presumably at file creation time rather than permanently, since ideally out of space is transient).  So we stop doing DOM entirely because we can't skip that one MDT.  That will require some interrogation of the MDTs but it should be possible, if a little awkward.

An alternate possibility occurs to me, sort of adjacent to the self-extending PFL work and exploiting similar ideas.  DOM is always the first component & so is always instantiated.  But we could do the "stripe to target, then check space on target before really making objects" trick I'm using in the SEPFL work (posting soon, sorry).

If there's insufficient space on a particular MDT, we could either try another MDT (...not so sure about this one...) or we could rewrite the layout dynamically to remove the DoM component, assuming it's followed by a regular OST component.  (Which it doesn't have to be.)

If we did the "try another MDT" solution, we'd either need to try all MDTs and fail once they were out of space, or we'd still have to do the "rewrite layout to replace DoM with regular OST layout" thing.

Just stop doing DoM once one MDT is full does seem like the easiest (and perhaps most comprehensible) behavior.  But there's another route if we wanted to take it.

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32482
Subject: LU-10808 lod: remove DoM component is disabled
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6b1cc1f9b42e8cacab90099c9c56f0ab96d9a471

[Mikhail Pershin]

Andreas wrote in patch comments:

What happens if we have a 16-way striped directory with a default layout like:

lfs setstripe -E 1M -L mdt -E 64M -c 1 -E -1 -c 128 /dir

and then one of the MDTs runs out of space? Ideally, it wouldn't run out of space (the MDS should reduce lod_dom_max_stripesize automatically as it gets full so the DoM components get smaller) but it will eventually run out of space. In case like this, we don't really want 1/16 of the files to return -EFBIG.

There is no way (currently) to disable file creation on a single MDT in the striped directory, so we would want to remove the DoM component from the file to reduce MDT space usage rather than failing file creation randomly based on the filename.

Is it possible to check if the layout is a default layout, or explicitly set by a user? It would be confusing if the user specifies a DoM component and it just silently disappears, but this is somewhat less of an issue if it is coming from a default setting (parent directory or filesystem root).

Put it here to don't forget and prepare workaround in later patches

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32073
Subject: LU-10808 lod: align wrong DoM stripe values with defaults
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 33206489d8b3252d96ca4d23d7774d54d8b085fc

[Mikhail Pershin]

yes, I found that too and already fixed this in a patch, it sets limit to the minimum rather than default value.

[Andreas Dilger]

I tested out setting dom_maxstripesize and if this is less than LOV_MIN_STRIPE_SIZE sets it to the default stripe size, rather than the minimum:

void lod_fix_desc_stripe_size(__u64 *val)
{               
        if (*val < LOV_MIN_STRIPE_SIZE) {
                if (*val != 0)
                        LCONSOLE_INFO("Increasing default stripe size to "
                                      "minimum value %u\n",
                                      LOV_DESC_STRIPE_SIZE_DEFAULT);
                *val = LOV_DESC_STRIPE_SIZE_DEFAULT;
        } else if (*val & (LOV_MIN_STRIPE_SIZE - 1)) {
                *val &= ~(LOV_MIN_STRIPE_SIZE - 1);
                LCONSOLE_WARN("Changing default stripe size to %llu (a "
                              "multiple of %u)\n",
                              *val, LOV_MIN_STRIPE_SIZE);
        }
}

Since this is also used for regular file striping (typically with stripe_size=0 to indicate the default) then it may be difficult to change this behaviour. One option is to pass in the "default" value from the caller, so that LOV can use LOV_DESC_STRIPE_SIZE_DEFAULT but DoM can use LOV_MIN_STRIPE_SIZE. Alternately, lod_dom_stripesize_seq_write() could check for a value below LOV_MIN_STRIPE_SIZE and return an error code to the caller.