Details
-
Improvement
-
Resolution: Unresolved
-
Minor
-
None
-
None
-
None
-
3
-
9223372036854775807
Description
Servers should be able to selectively allow client connections based on policies defined by Lustre admins. Policies could be defined on a wide range of client properties, depending on what proves to be most useful.
Attachments
Issue Links
- is related to
-
LU-15177 cs_update live batch update hung waiting for MDT recovery to complete
-
- Open
-
-
LU-13078 mgs trigger umount of clients
-
- Open
-
-
-
- Open
-
-
-
- Resolved
-
-
-
- Resolved
-
- is related to
-
-
- Open
-
Activity
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/52793/
Subject: LU-17217 obd: reserve server-side connection policy bits
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 55a65ed8853ba73f1f9de297e9b6e15a6f37743a
If a policy is set to SOFT_BLOCK, the client would be able to override by passing a mount flag (or something like that). The intended experience is like: client attempts to connect and is blocked with reason code -> user of client decides whether to proceed with mount with new information -> attempt connection again.
If the policy is set to HARD_BLOCK, perhaps we could hid the reason code. The client wouldn't be able to do anything. However, it'd still be nice to have a advisory message. Perhaps the advisory reason code could just be off by default.
The requirement about the ability for servers to provide a reason for blocking back to the client is interesting. Usually from a security perspective it is better to avoid explaining to clients the real reasons for a rejected access from the server. Not that server side is considered a black box, but because there is not really anything that the client can do. It is not like a badly formed IO that the application can fix for instance.
On the client-side, I think the important features (to me) are:
- SOFT_BLOCK functionality described above
- Ability for server to provide a reason for blocking back to the client
I think these would require some protocol change to work (specifically, those highlighted in the patch). In the case of a block, the reason code can be put in a unused part of the reply. So we should only need a few bit flags.
On the server side, we need to be able to set policies without knowing the NID (or much else about the client) ahead of time. In the PoC I have, this is done using custom commands and glob patterns. I could refactor it to either function on the default nodemap or add NID wildcards to nodemap. Then we could use nodemap modify commands with rbac or something.
I'm fine with refactoring all of the server-side stuff to fit more into nodemap. But I think the wire changes are needed regardless.
To me, this looks redundant in many aspects with what nodemap could offer, with the necessary extensions. The ability to identify clients based on their NIDs, the possibility to assign different properties to these clients, and then adapt server behavior based on these properties.
I really think we should not develop a whole new thing that is an equivalent of nodemap, but rather extend the nodemap capabilities to fit your needs. For instance the 'rbac' property of nodemap looks well suited for what you want, you would probably just have to introduce new values for this property.
Isn't the UUID decided by the client at mount time? How would the server know ahead of time what the UUID is? Perhaps I'm misunderstanding how that works.
yes, but it is constant across the lifetime of that client mount. So if a client mountpoint is suffering heartburn for some reason (LBUG while holding a DLM lock or mutex, whatever) then that UUID can be banned until it remounts. we don't necessarily need to ban the NID, since a reboot will fix the problem.
I'm not sure there could be any "free form" message unless a string was put in the server reply (in some place we don't care about because the connection was denied after all).
I don't think have a free-form message it too important personally. I think we should do only preset messages/codes unless someone has a strong argument for a wildcard/custom reason code.
For the LU-17435 case I think the server could use the same reason code but decide on the severity of the issue (e.g. 2-4 reconnects/resends per hour = WARN, 5-9 reconnects/resends per hour = SOFT_BLOCK, >= 10/hour = HARD_BLOCK), but that should be decided in the context of that ticket.
Are you suggesting a dynamic policy based on number of connection attempts? I could see that being useful. However, it would make defining the policies more complex. We'd probably have to use YAML rather than simple one-liners.
This makes me wonder if the SOFT_BLOCK mechanism should be on a "per reason code" basis, so that if the client admin says "sure I understand my client version is too old" and continues anyway, that doesn't give them free reign to connect if a new reason is added.
Agreed. When a client connects, we should scan all policies and apply the most strict one that matches. We can require each policy to associated with a reason code. That way, an admin can have multiple policies for each client property (if they want).
While the UUIDs would only be good for the duration of the mount instance, they would allow selectively denying one mountpoint on a client instead of all mountpoints.
Isn't the UUID decided by the client at mount time? How would the server know ahead of time what the UUID is? Perhaps I'm misunderstanding how that works.
I'm going to do a second revision of this PoC soon (I have other things I need to focus on first). I just want to make sure the design is more-or-less sketched out.
It would be very useful if this could be integrated with a policy like "deny clients by NID" independent of nodemaps, with a reason like "2:client NID administratively denied on server" or similar. Similar to "mdt.*.evict_nids" and "obdfilter.*.evict_nids" it should be possible to write client NIDs and maybe also UUIDs to the parameter to permanently block these connection attempts. While the UUIDs would only be good for the duration of the mount instance, they would allow selectively denying one mountpoint on a client instead of all mountpoints.
I was just looking into this in the context of LU-17435, and being able to dynamically add client NIDs to the "deny list" if they have issues maintaining a stable connection to the server. This would be managed by LNet/ptlrpc/ldlm in the kernel, so having a specific "reason code" would make sense, like "14:client has unstable network interface" or whatever, and a simpler check like "1:client Lustre version is too old" would probably be the first one. I'm not sure there could be any "free form" message unless a string was put in the server reply (in some place we don't care about because the connection was denied after all).
This makes me wonder if the SOFT_BLOCK mechanism should be on a "per reason code" basis, so that if the client admin says "sure I understand my client version is too old" and continues anyway, that doesn't give them free reign to connect if a new reason is added. I think that would mitigate my concerns with the usefulness of SOFT_BLOCK. For the LU-17435 case I think the server could use the same reason code but decide on the severity of the issue (e.g. 2-4 reconnects/resends per hour = WARN, 5-9 reconnects/resends per hour = SOFT_BLOCK, >= 10/hour = HARD_BLOCK), but that should be decided in the context of that ticket.
I'm thinking that the first time that SOFT_BLOCK is used (for whatever reason), the user would add the mount override option to the mount command line and then it would permanently ignore all future SOFT_BLOCK settings in the future.
Definitely. But I think that's fine. By then, the SOFT_BLOCK would have served it's social purpose. Plus, we can still have the client output some warning message. And the presence of the mount flag clearly indicates that the user has opted into an unusual (according to the admin) configuration.
Having a reason returned to the client would definitely be useful. If it was just the client version, then if the server returned a bogus higher version (e.g 215.3.0) and then the client would print a message like "Server testfs-MDT0000 version (215.3.0.0) is much newer than client. Consider upgrading client (2.12.9.0)".
I think the easiest way to do this: have the server return a code (defined by an enum) that the client can map to a nice message. I think that would need another u8 in obd_connect_data. So the total reservation would be: OBD_CONNECT flag, u8 bitmap (for SOFT_BLOCK, WARN flags), u8 for 256 server verdicts.
Sebastien, I was thinking about this a bit, and one reason to not use nodemaps for this is because nodemaps are relatively heavyweight to set up, and (IMHO) it would be useful to have a simple mechanism to allow blocking clients from accessing the servers. For example, if the servers are going into maintenance, or experiencing a problem due to some broken client application workload, it should be possible to quickly set a "deny all/some mounts" policy and evict clients, without having this be part of the persistent configuration that later has to be removed.
If there was a way to configure a nodemap temporarily like "lctl set_param ..." (vs. "lctl set_param -P") then that would be useful for this and likely other reasons we have discussed previously in LU-17431. We likely also need to have some way to configure (at least) a wildcard match for IPv6 addresses in a nodemap so that it could apply to all IPv6 nodes, which I've added to
LU-14288.