https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2898 Lustre <p>There have been periodic complaints about lustre not really knowing when it was evicted from a server node, as this could only be known in case an RPC is sent.<br/> Frequently this would be handled by a periodic ping, but with this functionality being turned down to happen in rarer cases, it more and more converts into the case of an app initiating an RPC and being evicted all of a sudden due to an eviction that has happened quite a while ago.</p> <p>As such we probably need a somewhat better way of notifying clients of their eviction so that they can reconnect somewhat more eagerly and with a bit less damage to whatever it is that might be running in userspace.</p> LU-2898

More timely notification of clients in case of eviction

Improvement Major Open Unresolved WC Triage Oleg Drokin Sun, 3 Mar 2013 23:51:19 +0000 Mon, 22 Jan 2024 15:57:16 +0000 0 7 <p>Fujitsu as the first site to disable pinging in most of the cases hit this esp. often so they created a patch to avert this issue that makes servers to notify MGS of eviction and MGS in turn would send messages to clients to come in contact with servers and reconnect as needed (sort of like reverse imperative recovery I guess).<br/> The contributed patch against fefs is here <a href="http://review.whamcloud.com/#change,5457" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5457</a> (And is not directly applicable to the master tree, but gives an idea of how they did it).</p> <p>I imagine it might have been much easier to just send a specially crafted ldlm callback to let it know we are evicting him (and this would require a lot less infrastructure changes), but that would not handle a case of severed communication between this particular server and client where as MGS connectivity of both would remain unaffected.</p> <p>Additionally since the case outlined as most severe by Fujitsu is that of a new application started, there is a possible workaround of doing "df" before a new job starts from whatever job scheduling framework might be there, but still there is a feeling that this case should be handled more transparently inside of Lustre.</p> <p>My first thought was that this does seem like a special case of imperative recovery, but limited to a specific client, and we could call it "imperative reconnect." <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> But perhaps a simpler ldlm callback is sufficient since if there is a network split the client wouldn't be able to reconnect anyway.</p> <p>Do we understand why these seemingly idle clients are being evicted in the first place? Is there an issue there?</p> <p>There might be many reasons for reconnect, I guess. All of them are valid one way or another. Like one-off AST loss or such.</p> <p>I agree that reconnects are probably valid, but I'm not sure all evicts are necessarily valid or unavoidable. If they are occurring frequently then we should at least try to find out what is causing them. </p> <p>Hi, Robert.</p> <p>I've often seen lots of clients are evicted when server recovering. It appeaers that a server cannot catch up with a great number of coming reconnect reqs, about 90k * (target-disks).<br/> As a reasult, clients whose recon reqs haven't been handled by the server are evicted.</p> <p>I see. Well, that's not ideal, but at least we know what the reason is. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>BTW, if the clients are not pinging, how did they all know to reconnect to the recovering server?</p> <p>Recovering serveres try to retrieve clients' information from last_rcvd files and see if they've been connected. And next, the serveres send callback pings to the clients in order to make them reconnect.<br/> this is a basic recovering motion in FEFS, thought lots of trivial<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/help_16.png" height="16" width="16" align="absmiddle" alt="" border="0"/> functions are included in it.</p> <p>oh, and I want you to know one thing, that is ... when we handle a large system like K, ping often eats up lnet resources such as credit<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/help_16.png" height="16" width="16" align="absmiddle" alt="" border="0"/> ... I'm not so good there, thought ... so I think you'll need a mesure against the problem. And which is why we restrict the retry number of times of callback ping to 5 times.</p> Related LU-2467 Development Rank 1|hzvk4f: Rank (Obsolete) 6985