https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16210 Lustre <p>On Centos7, selinux_is_enabled() was used to prevent this but on RHEL8 this interface disappears (on 5.1 kernel):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-c"> <span class="code-keyword">static</span> <span class="code-keyword"><span class="code-object">int</span></span> ll_xattr_get_common() ... <span class="code-comment">/* LU-549: Disable security.selinux when selinux is disabled */</span> if (handler-&gt;flags == XATTR_SECURITY_T &amp;&amp; !selinux_is_enabled() &amp;&amp; !strcmp(name, <span class="code-quote-red">"selinux"</span>)) RETURN(-EOPNOTSUPP); </pre> </div></div> <p>The patch <a href="https://review.whamcloud.com/38480" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38480</a> "<a href="https://jira.whamcloud.com/browse/LU-12355" title="Support for linux kernel version 5.0" class="issue-link" data-issue-key="LU-12355"><del>LU-12355</del></a> llite: include file linux/selinux.h removed" was submitted to add the compatibility to 5.1 kernel. But it considers selinux enabled if "selinux_is_enabled" is not found.</p> <p>So on RHEL8 with selinux disabled, we send unneeded RPCs on the MDS to retrieved security.selinux xattr. This has performance issues.</p> <p>Here the behavior with "ls":</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Centos 7 : getenforce 0 Disabled strace -e getxattr,lgetxattr ls -lda “directory” lgetxattr("directory", "security.selinux", 0x1096100, 255) = -1 EOPNOTSUPP (Operation not supported) getxattr("directory", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available) getxattr("directory", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available) Centos 8 : getenforce 0 Disabled lgetxattr("directory", "security.selinux", 0x1096100, 255) = -1 ENODATA (No data available) getxattr("directory", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available) getxattr("directory", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available) </pre> </div></div> <p>Here the "perf report" diff on getfattr:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Centos 7 : + 10.59% 0.00% getfattr libc-2.17.so [.] __GI___lxstat64 (inlined) + 10.59% 0.00% getfattr [kernel.vmlinux] [k] system_call + 10.59% 0.00% getfattr [kernel.vmlinux] [k] sys_newlstat + 10.59% 0.00% getfattr [kernel.vmlinux] [k] SYSC_newlstat + 10.59% 0.00% getfattr [kernel.vmlinux] [k] vfs_fstatat + 10.59% 0.00% getfattr [kernel.vmlinux] [k] user_path_at + 10.59% 0.00% getfattr [kernel.vmlinux] [k] user_path_at_empty + 10.59% 0.00% getfattr [kernel.vmlinux] [k] filename_lookup + 10.59% 0.00% getfattr [kernel.vmlinux] [k] path_lookupat + 10.59% 0.00% getfattr [kernel.vmlinux] [k] link_path_walk + 10.59% 0.00% getfattr [kernel.vmlinux] [k] inode_permission + 10.59% 0.00% getfattr [kernel.vmlinux] [k] __inode_permission + 10.59% 0.00% getfattr [lustre] [k] ll_inode_permission + 10.59% 0.00% getfattr [lustre] [k] ll_inode_revalidate + 10.59% 0.00% getfattr [lmv] [k] lmv_intent_lock Centos 8 : + 15.74% 0.00% getfattr libc-2.28.so [.] __GI_getxattr (inlined) + 15.74% 0.00% getfattr [kernel.kallsyms] [k] path_getxattr + 15.74% 0.00% getfattr [kernel.kallsyms] [k] getxattr + 15.74% 0.00% getfattr [kernel.kallsyms] [k] vfs_getxattr + 15.74% 0.00% getfattr [kernel.kallsyms] [k] __vfs_getxattr + 15.74% 0.00% getfattr [kernel.kallsyms] [k] ll_xattr_get_common + 15.74% 0.00% getfattr [kernel.kallsyms] [k] ll_xattr_list + 15.74% 0.00% getfattr [kernel.kallsyms] [k] lmv_getxattr + 15.74% 0.00% getfattr [kernel.kallsyms] [k] mdc_getxattr + 15.74% 0.00% getfattr [kernel.kallsyms] [k] mdc_xattr_common &lt;---------- + 15.74% 0.00% getfattr [kernel.kallsyms] [k] ptlrpc_queue_wait + 15.74% 0.00% getfattr [kernel.kallsyms] [k] ptlrpc_set_wait + 15.74% 0.00% getfattr [kernel.kallsyms] [k] ptlrpc_check_set.part </pre> </div></div> LU-16210

Lustre on RHEL8 with selinux disabled tries to retrieve security.selinux xattr

Bug Major Resolved Fixed Etienne Aujames Etienne Aujames SELinux performance Wed, 5 Oct 2022 09:13:30 +0000 Tue, 28 Mar 2023 13:38:17 +0000 Fri, 13 Jan 2023 14:49:20 +0000 Lustre 2.16.0 Lustre 2.15.3 0 7 <p>The CEA proposes to implement something like Ceph/NFS to fix this issue:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-c"> <span class="code-keyword"><span class="code-object">bool</span></span> ceph_security_xattr_wanted(<span class="code-keyword">struct</span> inode *in) { <span class="code-keyword">return</span> in-&gt;i_security != <span class="code-keyword">NULL</span>; } </pre> </div></div> <p>Thanks for reporting this.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Centos 7 : getenforce 0 Disabled strace -e getxattr,lgetxattr ls -lda “directory” lgetxattr("directory", "security.selinux", 0x1096100, 255) = -1 EOPNOTSUPP (Operation not supported) getxattr("directory", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available) getxattr("directory", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available) Centos 8 : getenforce 0 Disabled lgetxattr("directory", "security.selinux", 0x1096100, 255) = -1 ENODATA (No data available) getxattr("directory", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available) getxattr("directory", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available) </pre> </div></div> <p>Except the different return code (EOPNOTSUPP vs ENODATA), I do not see a different behavior between CentOS 7 and CentOS 8. Both ls are doing a getxattr on <tt>security.selinux</tt>, so what is the problem you are seeing?</p> <p>Hi Sebastien,</p> <p>The problem is, when ENODATA is reported, it means an action on the MDT side, but not when EOPNOTSUPP, the client will not try to fetch any security xattrs.</p> <p>In Etienne behavior description, `ls` check only the directory dentry, but with the full  `ls` of the directory, you can see :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>## CentOS 8 $ strace -e getxattr,lgetxattr ls -l lgetxattr("squashfs", "security.selinux", 0x564355468230, 255) = -1 ENODATA (No data available) getxattr("squashfs", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available) getxattr("squashfs", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available) lgetxattr("repository.app", "security.selinux", 0x564355477cc0, 255) = -1 ENODATA (No data available) getxattr("repository.app", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available) getxattr("repository.app", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available) lgetxattr("repository", "security.selinux", 0x564355477dd0, 255) = -1 ENODATA (No data available) getxattr("repository", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available) getxattr("repository", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available) lgetxattr("libpcocc_slurm.so", "security.selinux", 0x564355478cf0, 255) = -1 ENODATA (No data available) getxattr("libpcocc_slurm.so", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available) lgetxattr("ubuntu-mate", "security.selinux", 0x564355478e20, 255) = -1 ENODATA (No data available) getxattr("ubuntu-mate", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available) getxattr("ubuntu-mate", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available)   ## CentOS 7 $ strace -e getxattr,lgetxattr ls -l lgetxattr("squashfs", "security.selinux", 0xba0140, 255) = -1 EOPNOTSUPP (Operation not supported) getxattr("squashfs", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available) getxattr("squashfs", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available) getxattr("repository.app", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available) getxattr("repository.app", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available) getxattr("repository", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available) getxattr("repository", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available) getxattr("libpcocc_slurm.so", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available) getxattr("libpcocc_slurm.so", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available) getxattr("ubuntu-mate", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available) getxattr("ubuntu-mate", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available) </pre> </div></div> <p>So there are some optimisation in userspace saying that if a lgetxattr returns EOPNOTSUPP, don't try to gather selinux xattr anymore.</p> <p>There is no issue from users/applications standpoint.<br/> The issue here is performance: EOPNOTSUPP is returned directly from client llite layer, ENODATA is returned from MDT.<br/> That means we consume unnecessarily resources to send the RPC and get a reply. And "ll" by default try to get xattr "security.selinux".</p> <p>OK, it is clear to me now, thanks.</p> <p>I looked at how the <tt>i_security</tt> field is set in the kernel. Only SELinux sets it, via the following path:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>alloc_inode inode_init_always security_inode_alloc --&gt; sets i_security to NULL inode_alloc_security selinux_inode_alloc_security inode_alloc_security --&gt; sets i_security to non-NULL </pre> </div></div> <p>So indeed, we could consider that once inode has been allocated, <tt>i_security</tt> is not NULL if SELinux is enabled.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=eaujames" class="user-hover" rel="eaujames">eaujames</a> please add me as a reviewer to the patch you are cooking.</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48796" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48796</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16210" title="Lustre on RHEL8 with selinux disabled tries to retrieve security.selinux xattr" class="issue-link" data-issue-key="LU-16210"><del>LU-16210</del></a> llite: replace selinux_is_enabled()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b47eb08c932842af9aa10633b017e18153ad711d</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48875" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48875</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16210" title="Lustre on RHEL8 with selinux disabled tries to retrieve security.selinux xattr" class="issue-link" data-issue-key="LU-16210"><del>LU-16210</del></a> llite: replace selinux_is_enabled()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 19da032d77292cc068859e4dbe44a2376465515d</p> <p>*<a href="https://review.whamcloud.com/48796" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48796</a> : implements a solution with a simple i_security check.<br/> This will <b>only</b> work if SELinux is disabled at boot time (adding "selinux=0" on kernel cmdline) for RHEL &gt;= 8.5.<br/> *<a href="https://review.whamcloud.com/48875" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48875</a> : implements a more complex solution by retrieving and saving the security xattr name used by the LSM module at mount time (via security_inode_listsecurity()). Then it will use this stored value to filter out "security.selinux" (if SELinux is disabled, nothing is returned by security_inode_listsecurity()).<br/> This will work in <b>all</b> cases: SELinux is disabled via /etc/selinux/config or via kernel cmdline.</p> <p>Those patches restore the behavior of the <a href="http://review.whamcloud.com/2503" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2503</a> when SElinux is disabled:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 7cc542fd4c26ccb117ceb13a47ac8ced3107b9b3 Author: Yevheniy Demchenko &lt;zheka@uvt.cz&gt; Date: Tue Apr 10 22:01:14 2012 +0200 LU-549 llite: Improve statfs performance if selinux is disabled Even if selinux is disabled, client still tries to get selinux attributes from MDS. As xattrs are not yet cached, this significantly slows down xattr heavy operations like ls -l. This patch forces to return -EOPNOTSUPP on the client side if selinux is disabled. It speeds up ls -l 25% for cold-cache case and 50% for hot-cache case. </pre> </div></div> <p>Tests with "strace -c ls -l" with 100000 files on root in a multi VMs env (on Rocky 9). FS is remounted for each tests (cache is cleaned) and selinux is disabled.</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <th class='confluenceTh'>Total time %</th> <th class='confluenceTh'>lgetxattr</th> <th class='confluenceTh'>statx</th> </tr> <tr> <td class='confluenceTd'>Without the patch:</td> <td class='confluenceTd'>29%</td> <td class='confluenceTd'>51%</td> </tr> <tr> <td class='confluenceTd'>With the patch:</td> <td class='confluenceTd'>0%</td> <td class='confluenceTd'>87%</td> </tr> </tbody></table> </div> <p>"ls -l" uses lgetxattr to get "security.selinux".</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48875/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48875/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16210" title="Lustre on RHEL8 with selinux disabled tries to retrieve security.selinux xattr" class="issue-link" data-issue-key="LU-16210"><del>LU-16210</del></a> llite: replace selinux_is_enabled()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1d8faaf6caf4acaf0e2d4943b51c024a96c80624</p> <p>Landed for 2.16</p> <p>"Sebastien Buisson &lt;sbuisson@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49630" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49630</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16210" title="Lustre on RHEL8 with selinux disabled tries to retrieve security.selinux xattr" class="issue-link" data-issue-key="LU-16210"><del>LU-16210</del></a> llite: replace selinux_is_enabled()<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 06b50ff6c162ccdf57ecc59c81d8c266a184d582</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50404" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50404</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16210" title="Lustre on RHEL8 with selinux disabled tries to retrieve security.selinux xattr" class="issue-link" data-issue-key="LU-16210"><del>LU-16210</del></a> llite: replace selinux_is_enabled()<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 4ca0eceec6182e4c611d23c386c7e36c38d8e1fc</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49630/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49630/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16210" title="Lustre on RHEL8 with selinux disabled tries to retrieve security.selinux xattr" class="issue-link" data-issue-key="LU-16210"><del>LU-16210</del></a> llite: replace selinux_is_enabled()<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: 71143864865361d6b6012d75d8bb03a19a36b505</p> Development Rank 1|i0322n: Rank (Obsolete) 9223372036854775807 Severity