Also, though not a terrible defect, it could be beneficial to avoid prepping dentry and inode in ll_dir_setstripe() if SELinux is disabled.
2 points:
1) the review revealed the recovery defects which were not resolved before the land; we should try to avoid such a practice;
2) the patch suggests to use a not reliable security (in the recovery case); however, this is not about some attributes, this is about security, who wants to use a not reliable security?
=> should we probably revert a patch or document somewhere “the selinux support is not reliable, will be fixed in 2.9” would be enough ?
It should be noted that the items #2 and #3 are not new and a very similar issue is present in Android implementation.
The way they address it there is to have default context for all unlabelled files to be super restrictive such that nothing can access them. Then we allow the creator to change that to something else and make it actually accessible.
That way half-done objects are at least safe from other people accessing them.
This would need to be explained in documentation and all that of course.
My understanding is that the atomic setting of the SELinux xattr is something that Seagate/Xyratex has already implemented for SDA feature LU-5074, and we would welcome your contribution of those patches to the GPL LICENSED CODE THAT YOU ARE SHIPPING TO CUSTOMERS.
In the absence of those patches, we are moving forward with the SELinux functionality as time and resources permit, and it will very likely be incompatible with whatever you have currently implemented. If you aren't willing to contribute code that you already have to fix this, then please don't just provide objections to prevent it moving forward independently.
I don't see how SELinux support in SDA, if there is any, is related to this ticket. Intel blocked new connection flag landing when there was discussion on porting SDA code upstream in later development phases. The patch for the connection flag is still in gerrit, I believe. So, it's clearly Intel who broke compatibility, if there IS any SELinux support in SDA. (I'm not part of the SDA team and not aware of the current status of the development)
The reference to time and resources is weird. All defects listed above are known since September 2014, for 10 months already (issue 4 was described by the author in detail, others were described in gerrit). In order to help improve the code, I shared our experience from design phase in gerrit comments, though I wasn't formally asked to review the patch and had no formal obligation to comment it. I believe, only my comment about setstripe creation path was taken into account.
This ticket is not about preventing moving SELinux support forward. It just states what needs to be addressed to move forward and make the feature usable.
Actually, quite insignificant defects many times were the reason of rejecting a patch at the Intel side. However, this time there is tolerance and optimistic belief that there will be some movement forward, that we are preventing by just speaking of defects, after 10 months of not moving forward. Double standard?
Andreas,
may i remember you many my patches which you forbid to land because it isn't fully finished?
Why you land bad patched which needs for Intel and say "we are free to contribute fixes" and don't like to land other?
if feature isn't ready and produce a regressions - it feature should be forbid to land.. isn't ?
We expect a fair play if you are employed by OpenSFS to work as GateKepeer and do development for OpenSFS, but i don't see it for long time.
Intel blocked new connection flag landing when there was discussion on porting SDA code upstream in later development phases.
That is not accurate. On the contrary, Intel employees gave the patch positive reviews and were ready to land it. I was the person who blocked that patch. I am an employee of LLNL and a member of the leadership within OpenSFS.
I would like to try to now steer this conversation back to a discussion of technical merits.
I readily admit that I a may not have a deep enough understanding of the details yet to fully form a position. Nevertheless, Andrew's original point in the description section of this ticket seems valid to me, and worth more discussion.
Oleg's counter arguments that Android does the same thing and we can resolve the issue with documentation seem a bit weak to me.
First, Android distros control the entire OS environment. Lustre does not. Lustre only controls Lustre.
This design appears to be insecure by default, and it is up to the many system administrators to plug the hole in the design. The documentation of that fact will no doubt be burried in the giant operations manual, and I think it will be very common for system administrators to miss that requirement.
Even if all system administrators somehow manage to correctly configure their systems to be secure, any time this problem occurs it sounds like we will be left with files/directories that can only be cleaned up though administrative action. After faults, users will be left with files/directories that they cannot remove. This will generate more support tickets. The system administrators will probably not remember that the unremovable files are due to SELinux, so the problem will escalate to higher levels of support: local software developers or support vendors.
Our design should not add more work for support staff. Our design should not have defaults that are insecure and rely on human intervention to fix that.
I am still open to being convinced otherwise, but so far I am tending to agree that it was not yet ready for landing. This should probably be worked on a topic branch and reach a further level of completeness before merging into master.
Hi,
Enabling SELinux on a single node is in itself a source of complication for administrators: things that were just working out of the box without SELinux simply do not do anymore, and require fine security tuning to be fully functional again. Then one can imagine that enabling SELinux on a whole cluster necessarily implies administrative headaches. So I think we have to make clear that sites enabling SELinux on Lustre client nodes will inevitably see an increase in their support work. SELinux is complex and may require dedicated security administrators to maintain clusters in operational security conditions.
With appropriate documentation, the current implementation of SELinux support on Lustre client node could be considered as a tech preview with some restrictions.
This ticket could be the occasion to discuss the following technical points:
a) [atomicity]
b) [consistent file system view from different clients]
I consider [performance] and [recovery] will be solved when [atomicity] is addressed.
a) [atomicity]
In gerrit (http://review.whamcloud.com/11648), my proposition to address atomicity is to use a new security primitive (available from rhel7) named security_dentry_init_security(). It could be called from ll_lookup_it() as it only needs a dentry. Then security information could be sent with the lookup request (presumably creating a new request type and/or making a protocol change) so that the MDT can set the xattr in the same transaction as the file creation.
This proposition is only valid for rhel7, so far I cannot see an equivalent for rhel6.
b) [consistent file system view from different clients]
Coherency could be addressed thanks to a lock: setting or updating a security context would require a RW lock, and accessing a file would require a RO lock. What is unclear to me is the following: can we use for that an already existing lock type, or should we create a new lock type? Moreover, is there a description available somewhere about how to take and release locks in Lustre? or maybe some intelligible parts of the code that could be taken as an example to implement what we need here?
Of course, any technical remark or proposition would be much appreciated.
Thanks,
Sebastien.
Chris, I don't entirely agree with you about that my argument is weak.
(opensource) Lustre is a kit, but Andoird AOSP project as released by Google is also a kit. You do some legork to actually make that work on your device.
Or you can buy a premade device with the OS already tuned for it (from Google or whomever). And you can buy a Lustre appliance with the loose bits tied too.
Otherwise I feel we'll end up claiming that there's no failover in Lustre because you need to read documentation about how to do it and if you do it improperly - your filesystem would be damaged (doublemount and stuff). And perhaps other examples of the same (HSM is probably magnitudes more nuanced).
If you ask me, I feel that single client node SELinux support is in itself of a very limited use, it's like that lock that only keeps the honest people honest. Mount unauthorised or not-SELinux enabled or misconfigured client and the security is no more.
To have a robust solution we need to at least:
Enforce same policies on the servers (+protocol changes to actually transfer contexts around for every RPC - it's possible that existing bits there would somewhat work)
Authenticate clients and assign different roles (in SELinux policy speak) to different classes of clients (also some sort of kernel code signing?)
Disallow (or put in a separate role?) non-SELinux enabled clients.
Either replicate file contexts on OSTs (inflexible) or do access validation via MDS (more expensive) to avoid bad clients coming straight to OSTs and trying to read objects directly.
Do server validation (to ensure nobody registers phoney servers).
There are probably other must-haves, that I cannot think off the top of my head.
BTW forgot to add, despite all of the must-haves, even the limited support being added here has some uses as evidenced by people standing behind those patches and planning to use them.
It would probably aid the conversation to enumerate a use case or two. That might help people understand what exactly it is that we are getting with the current patch, and whether it made sense to land it as-is, or it should have waited for further work.
As it is, I suspect that quite of few people wishing to use the SELinux work may not understand the current patch's scope and limitations.
What the current patch does is to properly initiate the security context of a file created on an SELinux-enabled client, and store this security context on the server side (MDT) via the security.selinux extended attribute.
I can give one very simple example to help understand the utility of this.
Consider the home directories of the cluster's users. Every user has a .ssh directory in his/her home, containing a file with the list of authorized keys. When a remote connection is incoming, the sshd daemon checks the contents of this file to see if the incoming ssh key matches one referenced in the file.
When SELinux is enforced, the sshd daemon can only access the authorized keys file under ~/.ssh if this file has the ssh_home_t security context. If this file has no security context, or the default_t one, then one of the two following actions will be required in order to get ssh to work:
- the security administrator will have to relabel the file, so that the file gets the ssh_home_t security context; this action consists in scanning the file system;
- the security administrator will have to extend the SELinux policy so that sshd daemon is allowed to read a file that has the default_t security context; this action defeats the confinement concept that is the foundation of SELinux.
Local file systems like ext4 store files' security context on disk so that this information is made persistent.
Now consider that these home directories are located on a Lustre file system. By storing the files' security context permanently on the MDT, the current patch avoids, like a local file system, to resort to the two admin actions above.
Setting security context from client side has two major advantages:
- there is no need for SELinux on the server side: MDT only stores an extended attribute that is provided by the Lustre client;
- the cluster is simpler to administrate from a security point of view: all it needs is to have the same exact SELinux policy on every single Lustre client node.
On the opposite, setting security context on the server side would require to also have SELinux enforced on the MDS. And this would require an up-to-date policy equivalent to the one on the clients, but necessarily adapted because of the paths to the files in the MDT that differ from the ones on the clients that mount Lustre.
Protecting Lustre file systems from being mounted by SELinux-disabled clients is not SELinux' responsibility. Mechanisms like Kerberos or Shared Keys can address these problematics very well.
It shows that securing a cluster must be thought as a whole.
> What the current patch does is to properly initiate the security context of a file created on an SELinux-enabled client, and store this security context on the server side (MDT) via the security.selinux extended attribute.
Will don't work without selinux policy changes as Selinux use an xattr only for filesystems specially marked.
> Consider the home directories of the cluster's users. Every user has a .ssh directory in his/her home, containing a file with the list of authorized keys. When a remote connection is incoming, the sshd daemon checks the contents of this file to see if the incoming ssh key matches one referenced in the file.
Looks you forget one small note. if you enable an Selinux - it will affect to any files, not only ssh home where a static label exist.
so when any new file created it need assign a new label.
> On the opposite, setting security context on the server side would require to also have SELinux enforced on the MDS.
will not work, as selinux hooks called from VFS functions, but MDT/OST don't have such calls inside.
> Protecting Lustre file systems from being mounted by SELinux-disabled clients is not SELinux' responsibility. Mechanisms like Kerberos or Shared Keys can address these problematics very well.
It will done if you change a protocol correctly. Other pointed solutions not an address it problem, as knowledge about is client support selinux or not, just hide inside lustre and don't export to user land.
You forget about problem when different clients have a different Selinux polices ...
the cluster is simpler to administrate from a security point of view: all it needs is to have the same exact SELinux policy on every single Lustre client node.
Actually, this is kind of disadvantage of the approach. On the contrary, having policies applied on the MDS avoids any possible policy desync.
Sebastien, thank you for your example from July 10th.
It is unclear to me, though, what benefit is actually granted in your example versus standard posix file permissions. It sounds to me like this implements the SELinux API on the client, but fails to provide any actual security benefit. This sounds very dangerous. I suspect that most users and system administrators will simply hear that SELinux is now available in Lustre, and fail to recognize that it isn't really providing more security. That could put a lot of people at risk of failing security audits.
I have to agree with Andrew too, that the current design is a disadvantage, not a benefit. You are shifting security burden from the software onto human beings to correctly and uniformly configure all possible client nodes. Humans are error-prone. It also makes security auditing more difficult. Configuring the security policies once on the central server, if that is at all reasonable to do in software, sounds like a better approach.
Hi there,
Actually, this is kind of disadvantage of the approach. On the contrary, having policies applied on the MDS avoids any possible policy desync.
Do you mean that in the solution with policies applied on the MDS, SELinux is not enabled on the Lustre clients? In that case, how can the security policy be applied properly, without knowing the security context of the binary doing the action on Lustre?
Moreover, I would tend to think that security administrators that want SELinux for Lustre also want it for the root filesystem on their cluster's nodes. In other words, the customer approach is rather the following: now that SELinux is enabled on the cluster's nodes, how does it work when these nodes access Lustre?
In that case, having SELinux enforced on the MDS (requiring an up-to-date policy equivalent to the one on the clients) would represent an additional burden for security administrators.
It is unclear to me, though, what benefit is actually granted in your example versus standard posix file permissions.
The example of sshd and authorized keys file explains what is going on when a user connects via ssh to a node where SELinux is enabled. This is about making it work properly on Lustre when SELinux is enforced, hence getting the benefits of SELinux.
By definition, SELinux brings mandatory access-control policies that confine user programs and system servers access to files and network resources. Confinement is stronger than standard posix file permissions.
Of course, everyone willing to play with SELinux might be aware of what it is 
These issues have been resolved in other tickets.