[LU-10934] integrate statx() API with Lustre Created: 19/Apr/18  Updated: 14/Apr/22  Resolved: 27/May/20

Status: Resolved
Project: Lustre
Component/s: None
Affects Version/s: None
Fix Version/s: Lustre 2.14.0

Type: Improvement Priority: Minor
Reporter: Andreas Dilger Assignee: Qian Yingjin
Resolution: Fixed Votes: 0
Labels: medium

Issue Links:
Related
is related to LU-9538 Size on MDT with guarantee of eventua... Resolved
is related to LU-1158 nanosecond timestamp support for Lustre Reopened
is related to LU-11987 sanity test 59 fails with “test_59 fa... Open
is related to LU-13628 sanityn test_106a test_106c: supporte... Resolved
is related to LU-13826 fix compatibility for LL_IOC_MDC_GETINFO Resolved
is related to LU-10560 Fixes for 4.14 kernel Resolved
is related to LU-10805 fixes for 4.15 kernel Resolved
is related to LU-11916 LSOM: lazy_stat mount option not reco... Resolved
is related to LU-13148 Interop: sanity test 56ra fails with ... Resolved
is related to LU-13753 sanityn test_51b: 'file size is 1024,... Resolved
is related to LU-12480 add STATX_PROJID to upstream kernel Open
is related to LU-9558 Support linux kernel version 4.12 Resolved
is related to LU-11554 Make stat() work with LSOM Resolved
is related to LU-11971 Send file creation time to clients Resolved
is related to LU-12332 Add a liblustreapi call for IOC_MDC_G... Resolved
Rank (Obsolete): 9223372036854775807

 Description   

Kernels 4.10 and later (since commit a528d35e8b) have an enhanced statx() interface that can specify a bitmask to fetch specific attributes from a file (e.g. st_uid, st_gid, st_mode, and st_btime == file creation time), rather than fetching all of the normal stat() attributes (such as st_size and st_blocks). It also has a AT_STATX_DONT_SYNC mode which allows the kernel to return "lazy" stats without flushing all of the client data and fetching an accurate result from the server.

This matches (not coincidentally) very well with the Lustre attribute mechanism, and could be advantageous for fetching attributes from ls --color (needs st_mode) without having to do a glimpse to fetch the size (which is just discarded anyway).

The Lustre patch https://review.whamcloud.com/27715 "LU-9558 llite: user enhanced getattr functionality in newer kernels" added very basic support for this interface, but it only went so far as to handle the kernel API changes, but doesn't actually handle the different attribute flags appropriately (it currently fetches and returns all normal attributes - STATX_BASIC_STATS). This interface should be wired up properly so that we can take advantage of the efficiencies available.

  • only fetch MDS attributes if STATX_SIZE and STATX_BLOCKS are not requested, and avoid OSS glimpse RPCs completely
  • hook this into statahead to avoid async glimpse locks (AGL) if OST information not needed
  • enhance the MDS RPC interface to return the file creation time stored in both ldiskfs and ZFS already, and enable STATX_BTIME
  • in conjunction with Lazy Size-on-MDT (LSOM, LU-9538) return the LSOM size if AT_STATX_DONT_SYNC is used

 Comments   
Comment by Andreas Dilger [ 20/Mar/19 ]

The other question is whether ls uses statx() yet or not. Until that happens, there won't be any visible improvement to "ls --color" performance seen.

Comment by Aurelien Degremont (Inactive) [ 20/Mar/19 ]

As far as I've seen, ls guys are not really eager to support statx

Comment by Andreas Dilger [ 20/Mar/19 ]

I found that the stat command was patched to call statx() to fetch file creation time in coreutils 0.28, but unfortunately this is implemented by calling statx() again on the file, rather than calling statx() in the first place to fetch all of the required/available attributes. This means both stat() and statx() will be called on the file, increasing overhead rather than decreasing it . I didn't see other uses of statx() in the current coreutils.

Comment by Aurelien Degremont (Inactive) [ 21/Mar/19 ]

For Lustre the second call will likely just read data from cache. The performance impact should be low.

But this is not good news. Maybe we should push coreutils in the right direction.

 

Having a proper full statx implementation in Lustre is still useful. Robinhood can clearly benefit from it.

Comment by Andreas Dilger [ 11/Sep/19 ]

There is a very preliminary patch to add statx() support to ls at: https://github.com/jtlayton/coreutils/commits/statx

Comment by Andreas Dilger [ 11/Oct/19 ]

The support for stat(1) and ls(1) to use statx(3) to fetch only the required attributes has landed to the upstream GNU coreutils package.

Comment by Aurelien Degremont (Inactive) [ 28/Oct/19 ]

Nice Andreas!

Do you have a pointer for this change? Does this mean this is optimized and only statx() is used when possible?

 

Comment by Andreas Dilger [ 30/Oct/19 ]

The coreutils commit is at:
https://github.com/coreutils/coreutils/commit/a99ab266110795ed94a9cb4d2765ddad9c4310da

Comment by Gerrit Updater [ 05/Nov/19 ]

Yingjin Qian (qian@ddn.com) uploaded a new patch: https://review.whamcloud.com/36674
Subject: LU-10934 llite: integrate statx() API with Lustre
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 3bb2844b25767676ab56946a00aff16f02a82db9

Comment by Gerrit Updater [ 27/May/20 ]

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/36674/
Subject: LU-10934 llite: integrate statx() API with Lustre
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3f7853b31ef6f5a6c2837403910083057728a842

Comment by Peter Jones [ 27/May/20 ]

Landed for 2.14

Comment by Alex Zhuravlev [ 04/Jun/20 ]

tests sanityn/106* seem to be broken - even recent debugfs doesn't dump "btime".
probably crtime was meant?

Comment by Qian Yingjin [ 04/Jun/20 ]

Thanks for your report, Alex.

I will investigate it.

 

Regards,

Qian

Comment by Gerrit Updater [ 16/Jun/20 ]

Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/38947
Subject: LU-10934 tests: increase timeout for sanityn test_51b
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a00f3f63fa7e8d3ba9a32dcf4da3de83b6dcdcb3

Comment by Gerrit Updater [ 17/Jun/20 ]

Neil Brown (neilb@suse.de) uploaded a new patch: https://review.whamcloud.com/38973
Subject: LU-10934 tests: fix compilation without SELINUX
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6cdbb0e9730076cb24888dbae828ef52cc100082

Comment by Gerrit Updater [ 23/Jun/20 ]

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38973/
Subject: LU-10934 tests: fix compilation without SELINUX
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3aa5f8ecf31c29625057c19628d8bfbb3c0c26dc

Comment by Gerrit Updater [ 25/Aug/20 ]

Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/38947/
Subject: LU-10934 tests: increase timeout for sanityn test_51b
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9edec7ab92fb24f0a46e019c65a21481f9314d07

Comment by Andreas Dilger [ 04/Nov/20 ]

Support for ls with statx() is in coreutils-8.32-3.fc33 and later. It is disabled in coreutils-8.32-3.fc32 and earlier Fedora releases because of a bug in libseccomp2 (presumably explained and resolved in a private RH bug I don't have access to) that affects its use inside VMs running on RHEL7 and other hosts that do not have statx() system call).  Using this statx-enabled-ls should avoid the need for the client to get any OST attributes for the majority of files (i.e. those that have LSOM enabled and store the size on the MDT inode).

Generated at Sat Feb 10 02:39:28 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.