[LU-549] Add xattr list/value cache on client

• Status: Closed
• Project: Lustre
• Affects Version/s: Lustre 2.3.0
• Fix Version/s: Lustre 2.1.4
• Type: Bug
• Priority: Minor
• Reporter: Andreas Dilger
• Assignee: nasf (Inactive)
• Resolution: Duplicate
• Story Points: 5
• Severity: 3

Description

Caching the xattr list (and xattrs themselves) on the clients is becoming increasingly important, since this has shown up as a performance problem in the past (e.g. doing an RPC to get "security.capability" for every file access).

We can easily/compatibly add a simple xattr list and/or value cache on the client under the MDS_INODELOCK_UPDATE bit, which setxattr always gets in LCK_EX mode. mdt_reint_setxattr() also gets the MDS_INODELOCK_LOOKUP for XATTR_NAME_ACL_ACCESS, but this should be expanded to also include XATTR_SECURITY_PREFIX. This should also allow negative xattr caching so that an RPC is not sent repeatedly for xattrs that do not exist. Whether we want to cache the non-security xattrs under a separate MDS_INODELOCK_XATTR bit lock can be decided/implemented separately.

A second phase would be to read/send/cache common xattrs on the client on first access to avoid extra RPCs. This will become increasingly important with clients, since SELinux and other xattr-heavy tools are becoming increasingly common.

If we begin to cache xattrs on the client, we should remove the special-case handling of security.capability in ll_setxattr_common() which was added as a hack to address a serious performance problem.

Issue Links

• is related to: LU-2869 extended attribute cache for Lustre

Comments

[nasf (Inactive)]

The patch for packaged xattr:

http://review.whamcloud.com/#change,1212

[Andreas Dilger]

Fan Yong, do you have any performance tests to show what the improvement is with this patch?

[nasf (Inactive)]

I only tested "ls -l" for containing only sub-dirs case before, at least double the performance. Not test other cases yet.

[James A Simmons]

Against master this patch is broken. Can you update please.

[Peter Jones]

Fanyong

Let me know if someone else should take this work over

Peter

[nasf (Inactive)]

This is the patch against latest master:

http://review.whamcloud.com/#change,1212,set15

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,sles11,inkernel #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/llite/xattr.c
• lustre/autoconf/lustre-core.m4
• lustre/include/linux/lustre_compat25.h

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el5,inkernel #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/llite/xattr.c
• lustre/include/linux/lustre_compat25.h
• lustre/autoconf/lustre-core.m4

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el5,inkernel #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/llite/xattr.c
• lustre/autoconf/lustre-core.m4
• lustre/include/linux/lustre_compat25.h

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el6,ofa #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/autoconf/lustre-core.m4
• lustre/include/linux/lustre_compat25.h
• lustre/llite/xattr.c

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el5,ofa #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/llite/xattr.c
• lustre/autoconf/lustre-core.m4
• lustre/include/linux/lustre_compat25.h

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el5,ofa #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/include/linux/lustre_compat25.h
• lustre/autoconf/lustre-core.m4
• lustre/llite/xattr.c

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el6,inkernel #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/llite/xattr.c
• lustre/autoconf/lustre-core.m4
• lustre/include/linux/lustre_compat25.h

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el6,ofa #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/llite/xattr.c
• lustre/autoconf/lustre-core.m4
• lustre/include/linux/lustre_compat25.h

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el5,inkernel #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/include/linux/lustre_compat25.h
• lustre/autoconf/lustre-core.m4
• lustre/llite/xattr.c

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el5,inkernel #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/autoconf/lustre-core.m4
• lustre/llite/xattr.c
• lustre/include/linux/lustre_compat25.h

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el5,ofa #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/include/linux/lustre_compat25.h
• lustre/autoconf/lustre-core.m4
• lustre/llite/xattr.c

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el5,ofa #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/include/linux/lustre_compat25.h
• lustre/autoconf/lustre-core.m4
• lustre/llite/xattr.c

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el6,inkernel #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/include/linux/lustre_compat25.h
• lustre/autoconf/lustre-core.m4
• lustre/llite/xattr.c

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el6,inkernel #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/llite/xattr.c
• lustre/include/linux/lustre_compat25.h
• lustre/autoconf/lustre-core.m4

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el6,ofa #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/llite/xattr.c
• lustre/include/linux/lustre_compat25.h
• lustre/autoconf/lustre-core.m4

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el6,ofa #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/include/linux/lustre_compat25.h
• lustre/llite/xattr.c
• lustre/autoconf/lustre-core.m4

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el6,inkernel #498
LU-549 llite: Improve statfs performance if selinux is disabled (Revision 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3)

Result = SUCCESS
Oleg Drokin : 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Files :

• lustre/autoconf/lustre-core.m4
• lustre/include/linux/lustre_compat25.h
• lustre/llite/xattr.c

[Bob Glossman (Inactive)]

http://review.whamcloud.com/#change,2826
back port to b2_1

[James A Simmons]

Do we want this back ported to 2.1.X?

[James A Simmons]

Rebased to master at http://review.whamcloud.com/#change,1212

[Ian Colle (Inactive)]

See also http://review.whamcloud.com/#change,2503

[Andrew Perepechko]

Xyratex are implementing an xattr writethrough cache feature.
The patch has been coded and is currently under inspection.

An additional inode bit lock is introduced. A client
refills its xattr cache with a single RPC under LCK_PR
lock. setxattr RPC is sent synchronously and is updated
in the client cache under LCK_PW lock.

[Nathan Rutman]

Xyratex MRP-57

[Jodi Levi (Inactive)]

Duplicate of LU-2869