[LU-16210] Lustre on RHEL8 with selinux disabled tries to retrieve security.selinux xattr

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.16.0, Lustre 2.15.3
• Labels: SELinux performance
• Severity: 3
• Assignee: Etienne Aujames
• Reporter: Etienne Aujames

Description

On Centos7, selinux_is_enabled() was used to prevent this but on RHEL8 this interface disappears (on 5.1 kernel):

static int ll_xattr_get_common()
...
 	/* LU-549:  Disable security.selinux when selinux is disabled */
 	if (handler->flags == XATTR_SECURITY_T && !selinux_is_enabled() &&
	    !strcmp(name, "selinux"))
 		RETURN(-EOPNOTSUPP);

The patch https://review.whamcloud.com/38480 "LU-12355 llite: include file linux/selinux.h removed" was submitted to add the compatibility to 5.1 kernel. But it considers selinux enabled if "selinux_is_enabled" is not found.

So on RHEL8 with selinux disabled, we send unneeded RPCs on the MDS to retrieved security.selinux xattr. This has performance issues.

Here the behavior with "ls":

Centos 7 :
getenforce 0
Disabled
strace -e getxattr,lgetxattr ls -lda “directory”
lgetxattr("directory", "security.selinux", 0x1096100, 255) = -1 EOPNOTSUPP (Operation not supported)
getxattr("directory", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available)
getxattr("directory", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available)
 
Centos 8 :
getenforce 0
Disabled
lgetxattr("directory", "security.selinux", 0x1096100, 255) = -1 ENODATA (No data available)
getxattr("directory", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available)
getxattr("directory", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available)

Here the "perf report" diff on getfattr:

Centos 7 :
+   10.59%     0.00%  getfattr  libc-2.17.so      [.] __GI___lxstat64 (inlined)                                                                                            
+   10.59%     0.00%  getfattr  [kernel.vmlinux]  [k] system_call                                                                                                          
+   10.59%     0.00%  getfattr  [kernel.vmlinux]  [k] sys_newlstat                                                                                                         
+   10.59%     0.00%  getfattr  [kernel.vmlinux]  [k] SYSC_newlstat                                                                                                        
+   10.59%     0.00%  getfattr  [kernel.vmlinux]  [k] vfs_fstatat                                                                                                          
+   10.59%     0.00%  getfattr  [kernel.vmlinux]  [k] user_path_at                                                                                                         
+   10.59%     0.00%  getfattr  [kernel.vmlinux]  [k] user_path_at_empty                                                                                                   
+   10.59%     0.00%  getfattr  [kernel.vmlinux]  [k] filename_lookup                                                                                                      
+   10.59%     0.00%  getfattr  [kernel.vmlinux]  [k] path_lookupat                                                                                                        
+   10.59%     0.00%  getfattr  [kernel.vmlinux]  [k] link_path_walk                                                                                                       
+   10.59%     0.00%  getfattr  [kernel.vmlinux]  [k] inode_permission                                                                                                     
+   10.59%     0.00%  getfattr  [kernel.vmlinux]  [k] __inode_permission                                                                                                  
+   10.59%     0.00%  getfattr  [lustre]          [k] ll_inode_permission                                                                                                  
+   10.59%     0.00%  getfattr  [lustre]          [k] ll_inode_revalidate                                                                                                  
+   10.59%     0.00%  getfattr  [lmv]             [k] lmv_intent_lock  
 
Centos 8 :
+   15.74%     0.00%  getfattr  libc-2.28.so       [.] __GI_getxattr (inlined)                                                                                             
+   15.74%     0.00%  getfattr  [kernel.kallsyms]  [k] path_getxattr                                                                                                       
+   15.74%     0.00%  getfattr  [kernel.kallsyms]  [k] getxattr                                                                                                           
+   15.74%     0.00%  getfattr  [kernel.kallsyms]  [k] vfs_getxattr                                                                                                      
+   15.74%     0.00%  getfattr  [kernel.kallsyms]  [k] __vfs_getxattr                                                                                                      
+   15.74%     0.00%  getfattr  [kernel.kallsyms]  [k] ll_xattr_get_common                                                                                                 
+   15.74%     0.00%  getfattr  [kernel.kallsyms]  [k] ll_xattr_list                                                                                                       
+   15.74%     0.00%  getfattr  [kernel.kallsyms]  [k] lmv_getxattr                                                                                                        
+   15.74%     0.00%  getfattr  [kernel.kallsyms]  [k] mdc_getxattr                                                                                                        
+   15.74%     0.00%  getfattr  [kernel.kallsyms]  [k] mdc_xattr_common         <----------                                                                                      
+   15.74%     0.00%  getfattr  [kernel.kallsyms]  [k] ptlrpc_queue_wait                                                                                                   
+   15.74%     0.00%  getfattr  [kernel.kallsyms]  [k] ptlrpc_set_wait                                                                                  
+   15.74%     0.00%  getfattr  [kernel.kallsyms]  [k] ptlrpc_check_set.part

Comments

[Gerrit Updater]

"Etienne AUJAMES <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50404
Subject: LU-16210 llite: replace selinux_is_enabled()
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 4ca0eceec6182e4c611d23c386c7e36c38d8e1fc

[Gerrit Updater]

"Sebastien Buisson <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/49630
Subject: LU-16210 llite: replace selinux_is_enabled()
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: 06b50ff6c162ccdf57ecc59c81d8c266a184d582

[Peter Jones]

Landed for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/48875/
Subject: LU-16210 llite: replace selinux_is_enabled()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1d8faaf6caf4acaf0e2d4943b51c024a96c80624

[Etienne Aujames]

Those patches restore the behavior of the http://review.whamcloud.com/2503 when SElinux is disabled:

commit 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3
Author: Yevheniy Demchenko <[e-mail]>
Date:   Tue Apr 10 22:01:14 2012 +0200

    LU-549 llite: Improve statfs performance if selinux is disabled
    
    Even if selinux is disabled, client still tries to get selinux
    attributes from MDS. As xattrs are not yet cached, this significantly
    slows down xattr heavy operations like ls -l. This patch forces
    to return -EOPNOTSUPP on the client side if selinux is disabled.
    It speeds up ls -l 25% for cold-cache case and 50% for hot-cache
    case.

Tests with "strace -c ls -l" with 100000 files on root in a multi VMs env (on Rocky 9). FS is remounted for each tests (cache is cleaned) and selinux is disabled.

Total time %	lgetxattr	statx
Without the patch:	29%	51%
With the patch:	0%	87%

"ls -l" uses lgetxattr to get "security.selinux".

[Etienne Aujames]

*https://review.whamcloud.com/48796 : implements a solution with a simple i_security check.
This will only work if SELinux is disabled at boot time (adding "selinux=0" on kernel cmdline) for RHEL >= 8.5.
*https://review.whamcloud.com/48875 : implements a more complex solution by retrieving and saving the security xattr name used by the LSM module at mount time (via security_inode_listsecurity()). Then it will use this stored value to filter out "security.selinux" (if SELinux is disabled, nothing is returned by security_inode_listsecurity()).
This will work in all cases: SELinux is disabled via /etc/selinux/config or via kernel cmdline.

[Gerrit Updater]

"Etienne AUJAMES <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/48875
Subject: LU-16210 llite: replace selinux_is_enabled()
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 19da032d77292cc068859e4dbe44a2376465515d

[Gerrit Updater]

"Etienne AUJAMES <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/48796
Subject: LU-16210 llite: replace selinux_is_enabled()
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b47eb08c932842af9aa10633b017e18153ad711d

[Sebastien Buisson]

eaujames please add me as a reviewer to the patch you are cooking.

[Sebastien Buisson]

OK, it is clear to me now, thanks.

I looked at how the i_security field is set in the kernel. Only SELinux sets it, via the following path:

alloc_inode
   inode_init_always
      security_inode_alloc   --> sets i_security to NULL
         inode_alloc_security
            selinux_inode_alloc_security
               inode_alloc_security   --> sets i_security to non-NULL

So indeed, we could consider that once inode has been allocated, i_security is not NULL if SELinux is enabled.

[Etienne Aujames]

There is no issue from users/applications standpoint.
The issue here is performance: EOPNOTSUPP is returned directly from client llite layer, ENODATA is returned from MDT.
That means we consume unnecessarily resources to send the RPC and get a reply. And "ll" by default try to get xattr "security.selinux".