[LU-11376] Special file/dir to represent DAOS Containers

• Type: New Feature
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.13.0
• Labels: patch
• Assignee: Bruno Faccini (Inactive)
• Reporter: Johann Lombardi (Inactive)

Description

Add support for a new extended attribute or special file/dir layout to represent in the Lustre namespace an external DAOS container identified by a UUID. The pro’s and con’s of both options will be presented at the LAD’18 and summarized in this ticket.

The purpose of this ticket is to track the effort to implement this feature.

Issue Links

• is related to: LU-7207 HSM: Add Archive UUID to delete changelog records
• is related to: LU-12104 SEGV during "lfs find [--mdt-count=[+,-]<count>, --mdt-hash=<hashtype>]"
• is related to: LUDOC-465 Add feature documentation for foreign layout
• is related to: LU-10606 HSM info as part of LOV layout xattr
• is related to: LU-12682 foreign LOV/LMV format usage implementation for DAOS specific use case

Comments

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/34851
Subject: LU-11376 tests: skip sanity test_27K until it works
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b00742d9ca6b29dfd7d1de737d36dee3068ba995

[Peter Jones]

Landed for 2.13

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/34087/
Subject: LU-11376 lmv: new foreign LMV format
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: fdad38781cccb05c4cf3f1458c2d2d7c8b2b5bec

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/33755/
Subject: LU-11376 lov: new foreign LOV format
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6a20bdcc608bc2b933774b9f34ec25395e920a54

[Bruno Faccini (Inactive)]

In patch-set #10 of Gerrit Change #33755, new __u32 fields lmf_type and lmf_flags have been added to foreign LOV format.

In patch-set #8 of Gerrit Change #34087, new __u32 fields lmf_type and lmf_flags have been added to foreign LMV format.

 

[Bruno Faccini (Inactive)]

Nathan, Andreas, this sounds reasonable, I will add those fields now.

[Andreas Dilger]

I think the addition of two fields into the data structure for forward compatibility is not a huge burden, and I agree with Nathan that this is easily done now and harder to do later. Using those fields for various purposes can be deferred to later patches when that functionality is needed.

[Nathan Rutman]

I would rather address those changes in follow-on patches.

I get the desire to both keep it simple and minimize churn, but the (potential) price to pay is future code complexity for compat checking. Since this is an on-disk structure, we will bear that burden into the eternal future. It's very cheap to do this now (literally, just add the fields - don't even need to bother with swapping them today), very expensive to do it later. Are there concrete plans to do something with these new fields? No. But it seems like it could have value in letting Lustre make some decisions about where/when/how such external files are handled.
@adilger do you want to weigh in?

[Johann Lombardi (Inactive)]

Thanks Nathan. Unless this is a requirement to land this patch, I would rather address those changes in follow-on patches.

[Nathan Rutman]

/* foreign LMV EA */
struct lmv_foreign_md {
	__u32 lfm_magic;	/* magic number = LOV_MAGIC_FOREIGN */
	__u32 lfm_length;	/* length of lfm_value */
	char lfm_value[];
};

Can we add fields here for __u32 lfm_type and __u32 lfm_flags so that Lustre can in the future have some way of understanding the contents? If everything is hidden inside lfm_value, then Lustre would have no way of, for example:

• selecting a client-side handler/upcall for different foreign types
• selecting a server-side handler of any type
• invoking the HSM mechanisms to automatically populate/swap-in a file