[LU-11367] integrate LSOM with lfs find

• Type: Improvement
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.13.0, Lustre 2.12.4
• Affects Version/s: Lustre 2.12.0
• Labels: LSOM performance
• Assignee: Qian Yingjin
• Reporter: Andreas Dilger

Description

The LSOM functionality should be integrated with lfs find so that it is possible to use this functionality directly on the client. The MDS can fill in the mbo_size and mbo_blocks fields from the LSOM xattr, if the actual size/blocks are not available, and then set new OBD_MD_FLLSIZE and OBD_MD_FLLBLOCKS flags in the reply so that the client knows these fields are valid.

The lfs find command can add a "--lazy" option (or maybe "--lsize" and "--lblocks"?) to allow the use of LSOM data from the MDS. We might consider to also use this for "--size" when it is specified in some large range, like "--size -1G" or "--blocks +1M" since we don't need accurate sizes to match those requests.  That could probably be done as a separate patch after the initial functionality is implemented and we have some idea how well LSOM is keeping the size uptodate.

Please include updates to the command usage and man page.

Issue Links

• is related to: LU-13826 fix compatibility for LL_IOC_MDC_GETINFO
• is related to: LU-12337 SoM in DoM needs to work transparently
• is related to: LU-12026 verify that MDS stores atime/mtime/ctime during LSOM update
• is related to: LU-11554 Make stat() work with LSOM
• is related to: LU-11971 Send file creation time to clients
• is related to: LU-12332 Add a liblustreapi call for IOC_MDC_GETFILEINFO
• is related to: LU-12253 lfs: double free or corruption (!prev)
• is related to: LU-9538 Size on MDT with guarantee of eventual consistency

Comments

[John Hammond]

Renaming ioctls in this way is a bad idea and will only cause more problems than it fixes. Please do not do this.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/36553/
Subject: LU-11367 som: integrate LSOM with lfs find
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: d55982d8422abada4e9f23856e18d20771f70897

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36553
Subject: LU-11367 som: integrate LSOM with lfs find
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 982f676adb37f77e4dcf067684753ef88a9aab76

[Peter Jones]

Landed for 2.13

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35167/
Subject: LU-11367 som: integrate LSOM with lfs find
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 11aa7f8704c490b011f60f234c3ac9929ce76948

[Andreas Dilger]

It would be useful, but not required before the patch is landed, to run a performance comparison of "lfs find $MOUNT --lazy -size +N" with "find $MOUNT -size +N", clearing the cache before the two runs, especially if the OSTs are disk based and the MDT is flash based (or cached in RAM). That would be useful to include in the commit message (if ready in time), and for presentations about this feature.

[Nathan Rutman]

A related ticket added llapi accessor functions for lum components, but I can't seem to find it now - if someone knows, please add an issue link.

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35167
Subject: LU-11367 som: integrate LSOM with lfs find
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f05a7569e705abd6b3d9f458ae3a7c002c87248a

[Andreas Dilger]

We need a new version of patch https://review.whamcloud.com/33545 pushed with the fix for LU-12253.

[Oleg Drokin]

reopening because of revert.