https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5703 Lustre <p>In order to minimize user disruptions NASA performs some system maintenance "Live". Typical maintance includes activities such as adding new compute node or reconfigurations of IB fabric. During such times users jobs are suspend via pbs. Although we are able to suspend user job, which does minimize usage of lustre, it does not stop all lustre client/server activity. Therefore NASA requires:<br/> 1. mechanism to halt and block all lustre client IO.<br/> 2. Halt client/server keep alive ping and all other network traffic.<br/> 3. Clients should be able to recover after the quiesce without eviction.</p> LU-5703

Quiesce client mountpoints from the server

New Feature Major Open Unresolved Peter Jones Mahmoud Hanafi Thu, 2 Oct 2014 17:13:49 +0000 Thu, 18 Nov 2021 22:33:30 +0000 Lustre 2.4.3 1 14 <p>The simplest way to do this would be to umount the clients, Then remount after your maintenance. That leaves the client in a good state, and you should be able to restart without issue. <br/> If you are adding new compute nodes, (not servers) that should be completely transparent to all other clients, and should never require any quiescing of Lustre. Clients are very independent of one another. </p> <p>If you are changing IB values, it would be best to umount all Lustre, unload the LNET modules and restart. That way you are certain your IB changes would propagate. </p> <p>In most cases of Lustre 'live' maintenance, any live but idle Lustre machines should cause you no issues. There is no need for a 'quiesce' If you need to completely eliminate all Lustre traffic from your network, the quickest and safest way to do this is to simply stop Lustre on the affected nodes. </p> <p>Cliff, I expect there may be problems relating to open files on the mounted filesystem that cannot be closed without killing the running application.</p> <p>This request sounds a lot like a requirement we had for a feature called "Simplified Interoperability", which would flush all uncommitted RPCs from the clients and quiesce them in advance of a server shutdown for a Lustre software upgrade, so that we didn't have to manage recovery/replay of Lustre RPCs across different versions. This requires work on both the clients (to be able to "detach" themselves from their open files and "reattach" them once the server is restarted), and on the servers to notify the clients of the impending shutdown and to allow the clients to reconnect without evicting them.</p> <p>At NCSA, there is a similar need although possibly lighter weight than the Simplified Interoperability feature described, and possibly just items 1 &amp; 3 described by Mahmoud. <br/> The specific application I have in mind at the moment is for confirmation benchmarking after an online configuration tuning. Ideally in this case, all clients would remain actively mounted (and pinging) with existing open files, but would suspend operations beyond that. A /proc control on the client would tell it whether or not to "suspend" or not, which then leaves the capability to have some clients active (presumably used to execute the regression test).<br/> A search for this capability landed this ticket as a result- perhaps it's different enough that I should open a separate RFE?</p> <p>Is this request for maintenance capabilities about server changes only?  That is, the request isn't about keeping files open while the Lustre client is changed, correct?</p> <p>I think it's about client and configuration changes first... if server changes were possible too, that'd be outstanding. It is <b>not</b> about changing config or versions on clients in use by jobs while holding files open.  The idea is to idle the clients in use by jobs so that <em>others</em> in a test pool could run regression tests after a config or client change, or regression tests on a periodic basis.</p> <p>Getting consistent, and therefore meaningful, regression tests w/o a dedicated system is impossible otherwise.</p> Duplicate LU-13078 Related LU-13010 LU-18 LU-7236 LU-3290 LU-13521 LU-15250 Development End date Thu, 26 Feb 2015 17:13:49 +0000 Rank 1|hzwxnj: Rank (Obsolete) 15964 Start date Thu, 2 Oct 2014 17:13:49 +0000