[LU-5703] Quiesce client mountpoints from the server

• Type: New Feature
• Resolution: Unresolved
• Priority: Major
• Affects Version/s: Lustre 2.4.3
• Labels: medium
• Assignee: Peter Jones
• Reporter: Mahmoud Hanafi

Description

In order to minimize user disruptions NASA performs some system maintenance "Live". Typical maintance includes activities such as adding new compute node or reconfigurations of IB fabric. During such times users jobs are suspend via pbs. Although we are able to suspend user job, which does minimize usage of lustre, it does not stop all lustre client/server activity. Therefore NASA requires:
1. mechanism to halt and block all lustre client IO.
2. Halt client/server keep alive ping and all other network traffic.
3. Clients should be able to recover after the quiesce without eviction.

Issue Links

• is duplicated by: LU-13078 mgs trigger umount of clients
• is related to: LU-3290 disallow ptlrpc RPCs with old client XIDs
• is related to: LU-13521 WBC: special readdir() handling for root WBC directory
• is related to: LU-15250 RPC Replay Signature
• is related to: LU-13010 WBC: Reopen the file when WBC EX lock revoking
• is related to: LU-18 Allow 100k open files on single client
• is related to: LU-7236 OST connect and disconnect on demand

Comments

[Andreas Dilger]

Related to Patrick's recent comment I've attached Dynamic Congestion Control - Qian.docx, which is a paper that Yingjin worked on long ago to have the servers dynamically manage the client max_rpcs_in_flight, in a manner similar to grants (i.e. constant flow of "RPC credits" out to clients that need them and withdrawing them from other clients that are not active). In addition to "stop all client RPCs" this could also be used to temporarily boost performance to busy clients.

[Patrick Farrell]

I think this could probably be done fairly easily with some tweaks to the RPC code to allow the server to temporarily set max_rpcs_in_flight for data and metadata to zero.  It wouldn't guarantee total silence - that would require unmounting - but it might be useful if folks didn't mind taking an additional performance stoppage.  Interesting.