[LU-6784] Defects in SELinux support

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Affects Version/s: Lustre 2.8.0
• Severity: 3
• Assignee: WC Triage
• Reporter: Andrew Perepechko

Description

I was asked to file a ticket and summarize defects in the landed implementation of SELinux support which had been reported prior to landing:

1) [performance] create and setxattr are separate RPCs which makes things slow
2) [recovery] create and setxattr are separate RPCs, if the client crashes in between of create and setxattr, the file will not get a security label; client kernels will use default security labels in this case, but default label concept is not to work around file system bugs and no default label will be consistent with SELinux security model
3) [atomicity] create and setxattr are separate RPCs, if another client accesses the same file, it will see no security label, it will raise the same issues as in (2)
4) [consistent file system view from different clients] although initial versions of the landed patch made attempts to synchronize relabel operations among clients, the final patch does not implement any synchronization, so inodes in memory will keep old security labels in inode->i_security (this is documented in LU-5560)

Issue Links

• is blocking: LU-5560 SELinux support on the client side
• is related to: LU-9193 Multiple hangs observed with many open/getattr
• is related to: LU-13742 Provide an selinux_is_enabled for newer kernels
• is related to: LU-5074 Connection flag reservation for Secure Data Appliance
• is related to: LU-8955 Send SELinux policy info to server

Comments

[Sebastien Buisson (Inactive)]

What the current patch does is to properly initiate the security context of a file created on an SELinux-enabled client, and store this security context on the server side (MDT) via the security.selinux extended attribute.

I can give one very simple example to help understand the utility of this.
Consider the home directories of the cluster's users. Every user has a .ssh directory in his/her home, containing a file with the list of authorized keys. When a remote connection is incoming, the sshd daemon checks the contents of this file to see if the incoming ssh key matches one referenced in the file.
When SELinux is enforced, the sshd daemon can only access the authorized keys file under ~/.ssh if this file has the ssh_home_t security context. If this file has no security context, or the default_t one, then one of the two following actions will be required in order to get ssh to work:

• the security administrator will have to relabel the file, so that the file gets the ssh_home_t security context; this action consists in scanning the file system;
• the security administrator will have to extend the SELinux policy so that sshd daemon is allowed to read a file that has the default_t security context; this action defeats the confinement concept that is the foundation of SELinux.
  Local file systems like ext4 store files' security context on disk so that this information is made persistent.

Now consider that these home directories are located on a Lustre file system. By storing the files' security context permanently on the MDT, the current patch avoids, like a local file system, to resort to the two admin actions above.

Setting security context from client side has two major advantages:

• there is no need for SELinux on the server side: MDT only stores an extended attribute that is provided by the Lustre client;
• the cluster is simpler to administrate from a security point of view: all it needs is to have the same exact SELinux policy on every single Lustre client node.
  On the opposite, setting security context on the server side would require to also have SELinux enforced on the MDS. And this would require an up-to-date policy equivalent to the one on the clients, but necessarily adapted because of the paths to the files in the MDT that differ from the ones on the clients that mount Lustre.

Protecting Lustre file systems from being mounted by SELinux-disabled clients is not SELinux' responsibility. Mechanisms like Kerberos or Shared Keys can address these problematics very well.
It shows that securing a cluster must be thought as a whole.

[Christopher Morrone (Inactive)]

It would probably aid the conversation to enumerate a use case or two. That might help people understand what exactly it is that we are getting with the current patch, and whether it made sense to land it as-is, or it should have waited for further work.

As it is, I suspect that quite of few people wishing to use the SELinux work may not understand the current patch's scope and limitations.

[Oleg Drokin]

BTW forgot to add, despite all of the must-haves, even the limited support being added here has some uses as evidenced by people standing behind those patches and planning to use them.

[Oleg Drokin]

Chris, I don't entirely agree with you about that my argument is weak.

(opensource) Lustre is a kit, but Andoird AOSP project as released by Google is also a kit. You do some legork to actually make that work on your device.
Or you can buy a premade device with the OS already tuned for it (from Google or whomever). And you can buy a Lustre appliance with the loose bits tied too.

Otherwise I feel we'll end up claiming that there's no failover in Lustre because you need to read documentation about how to do it and if you do it improperly - your filesystem would be damaged (doublemount and stuff). And perhaps other examples of the same (HSM is probably magnitudes more nuanced).

If you ask me, I feel that single client node SELinux support is in itself of a very limited use, it's like that lock that only keeps the honest people honest. Mount unauthorised or not-SELinux enabled or misconfigured client and the security is no more.
To have a robust solution we need to at least:
Enforce same policies on the servers (+protocol changes to actually transfer contexts around for every RPC - it's possible that existing bits there would somewhat work)
Authenticate clients and assign different roles (in SELinux policy speak) to different classes of clients (also some sort of kernel code signing?)
Disallow (or put in a separate role?) non-SELinux enabled clients.
Either replicate file contexts on OSTs (inflexible) or do access validation via MDS (more expensive) to avoid bad clients coming straight to OSTs and trying to read objects directly.
Do server validation (to ensure nobody registers phoney servers).

There are probably other must-haves, that I cannot think off the top of my head.

[Sebastien Buisson (Inactive)]

Hi,

Enabling SELinux on a single node is in itself a source of complication for administrators: things that were just working out of the box without SELinux simply do not do anymore, and require fine security tuning to be fully functional again. Then one can imagine that enabling SELinux on a whole cluster necessarily implies administrative headaches. So I think we have to make clear that sites enabling SELinux on Lustre client nodes will inevitably see an increase in their support work. SELinux is complex and may require dedicated security administrators to maintain clusters in operational security conditions.
With appropriate documentation, the current implementation of SELinux support on Lustre client node could be considered as a tech preview with some restrictions.

This ticket could be the occasion to discuss the following technical points:
a) [atomicity]
b) [consistent file system view from different clients]
I consider [performance] and [recovery] will be solved when [atomicity] is addressed.

a) [atomicity]
In gerrit (http://review.whamcloud.com/11648), my proposition to address atomicity is to use a new security primitive (available from rhel7) named security_dentry_init_security(). It could be called from ll_lookup_it() as it only needs a dentry. Then security information could be sent with the lookup request (presumably creating a new request type and/or making a protocol change) so that the MDT can set the xattr in the same transaction as the file creation.
This proposition is only valid for rhel7, so far I cannot see an equivalent for rhel6.

b) [consistent file system view from different clients]
Coherency could be addressed thanks to a lock: setting or updating a security context would require a RW lock, and accessing a file would require a RO lock. What is unclear to me is the following: can we use for that an already existing lock type, or should we create a new lock type? Moreover, is there a description available somewhere about how to take and release locks in Lustre? or maybe some intelligible parts of the code that could be taken as an example to implement what we need here?

Of course, any technical remark or proposition would be much appreciated.
Thanks,
Sebastien.

[Christopher Morrone (Inactive)]

I would like to try to now steer this conversation back to a discussion of technical merits.

I readily admit that I a may not have a deep enough understanding of the details yet to fully form a position. Nevertheless, Andrew's original point in the description section of this ticket seems valid to me, and worth more discussion.

Oleg's counter arguments that Android does the same thing and we can resolve the issue with documentation seem a bit weak to me.

First, Android distros control the entire OS environment. Lustre does not. Lustre only controls Lustre.

This design appears to be insecure by default, and it is up to the many system administrators to plug the hole in the design. The documentation of that fact will no doubt be burried in the giant operations manual, and I think it will be very common for system administrators to miss that requirement.

Even if all system administrators somehow manage to correctly configure their systems to be secure, any time this problem occurs it sounds like we will be left with files/directories that can only be cleaned up though administrative action. After faults, users will be left with files/directories that they cannot remove. This will generate more support tickets. The system administrators will probably not remember that the unremovable files are due to SELinux, so the problem will escalate to higher levels of support: local software developers or support vendors.

Our design should not add more work for support staff. Our design should not have defaults that are insecure and rely on human intervention to fix that.

I am still open to being convinced otherwise, but so far I am tending to agree that it was not yet ready for landing. This should probably be worked on a topic branch and reach a further level of completeness before merging into master.

[Christopher Morrone (Inactive)]

Intel blocked new connection flag landing when there was discussion on porting SDA code upstream in later development phases.

That is not accurate. On the contrary, Intel employees gave the patch positive reviews and were ready to land it. I was the person who blocked that patch. I am an employee of LLNL and a member of the leadership within OpenSFS.

[Alexey Lyashkov]

We expect a fair play if you are employed by OpenSFS to work as GateKepeer and do development for OpenSFS, but i don't see it for long time.

[Alexey Lyashkov]

Andreas,

may i remember you many my patches which you forbid to land because it isn't fully finished?
Why you land bad patched which needs for Intel and say "we are free to contribute fixes" and don't like to land other?
if feature isn't ready and produce a regressions - it feature should be forbid to land.. isn't ?

[Andrew Perepechko]

I don't see how SELinux support in SDA, if there is any, is related to this ticket. Intel blocked new connection flag landing when there was discussion on porting SDA code upstream in later development phases. The patch for the connection flag is still in gerrit, I believe. So, it's clearly Intel who broke compatibility, if there IS any SELinux support in SDA. (I'm not part of the SDA team and not aware of the current status of the development)

The reference to time and resources is weird. All defects listed above are known since September 2014, for 10 months already (issue 4 was described by the author in detail, others were described in gerrit). In order to help improve the code, I shared our experience from design phase in gerrit comments, though I wasn't formally asked to review the patch and had no formal obligation to comment it. I believe, only my comment about setstripe creation path was taken into account.

This ticket is not about preventing moving SELinux support forward. It just states what needs to be addressed to move forward and make the feature usable.

Actually, quite insignificant defects many times were the reason of rejecting a patch at the Intel side. However, this time there is tolerance and optimistic belief that there will be some movement forward, that we are preventing by just speaking of defects, after 10 months of not moving forward. Double standard?

[Andreas Dilger]

My understanding is that the atomic setting of the SELinux xattr is something that Seagate/Xyratex has already implemented for SDA feature LU-5074, and we would welcome your contribution of those patches to the GPL LICENSED CODE THAT YOU ARE SHIPPING TO CUSTOMERS.

In the absence of those patches, we are moving forward with the SELinux functionality as time and resources permit, and it will very likely be incompatible with whatever you have currently implemented. If you aren't willing to contribute code that you already have to fix this, then please don't just provide objections to prevent it moving forward independently.