https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10275 Lustre <p>Because most ptlrpc messages do not have ACK , RPC client cannot distinguish message loss from long service time. Also, in current implementation, message-resend can only be triggered by RPC client after service timeout, no matter which message is lost in lifecycle of RPC.</p> <p>To improve Lustre RAS against message loss, we should allow message resend for any step of RPC lifecycle. However, current RPC client already has request message timeout/resend protocol and adaptive timeout, it may need fundamental changes if we want to have ACK for request message and use network timeout instead of service time to trigger request message resend. This may require a lot more efforts and resources, so it is not covered by this document.</p> <p>Reply-resend is relatively simple and more practicable, RPC server can repeatedly resend reply at fix time interval (e.g. 20 seconds), which should be sufficient even for latency in environment with router. Reply-resend can be stopped when there is an ACK for reply message, or client is evicted/disconnected.</p> LU-10275

ptlrpc reply acknowledgement

New Feature Minor Open Unresolved Amir Shehata Liang Zhen lnet performance Sun, 21 Dec 2014 07:00:49 +0000 Tue, 18 Dec 2018 02:58:06 +0000 0 8 <p>I created a dedicated patch for lnet event add-on: <a href="https://jira.whamcloud.com/browse/LU-10274" title="LNet event add-on function" class="issue-link" data-issue-key="LU-10274"><del>INTL-166</del></a></p> <p>patch list:<br/> <a href="http://review.whamcloud.com/#/c/13203" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13203</a><br/> <a href="http://review.whamcloud.com/#/c/13204" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13204</a><br/> <a href="http://review.whamcloud.com/#/c/13219" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13219</a><br/> <a href="http://review.whamcloud.com/#/c/13220" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13220</a><br/> <a href="http://review.whamcloud.com/#/c/13227" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13227</a><br/> <a href="http://review.whamcloud.com/#/c/13228" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13228</a><br/> <a href="http://review.whamcloud.com/#/c/13489" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13489</a></p> <p>Liang, have you done any testing on this to determine how much it improves reliability, recoverability, etc? Any idea on what kind of performance impact it has on normal operation? What kind of interoperability is needed for this (i.e. can it be done on a per-client basis, or do all clients need it, or what)?</p> <p>Andreas, because this patch can only improve reliability on one direction so far, so I did some tests with enabling message drop for reply portals only, without this patch, I got client eviction time to time while running random workload. After I applied this patch (because it is a small cluster, so I set reply-resend interval to a small value like 2-4 seconds instead of default value, so it can resend within AT), there was almost no client eviction.Of course we can't expect improvement like this in real world because message drop can be on both directions, but I think it is a good step to start.<br/> This feature can be applied on a per-client basis, or let's say we can upgrade any node without breaking interoperability, this feature is enabled only when both end of ptlrpc connection has this patch. Also, it can be enabled/disabled at runtime.</p> <p>I will attach some data, these data are not from this patch because I collected them before I worked this patch, but from a simple patch to enable ACK for all ptlrpc messages, so I think they are essentially same. From these data, we lost about 10% performance for lightweight metadata operations (0-stripe) when we have ACK for all messages (both request &amp; reply), so I assume we may lose 5% performance with reply-ack only.</p> <p>performance data for ACKed lnet messages.</p> <p>Andreas, do you have any concern/comment on these data? </p> <p>Liang, thanks for the data. It looks like the overhead is noticeable, but not so bad that the change is unusable.</p> <p>Is it possible to make this feature optional, so that we can turn it on or off to debug?</p> <p>Andreas, yes I think I can do this, I will update the patch to make it optional.</p> <p>I will work on CORAL soon, so have to reassign this ticket to bobijam. I will maintain patches for a while, but can't finish the landing process.</p> <p>Amir, how does this relate to our recent discussions about LNet Health and reply timeouts? Are these patches still useful?</p> <p>This looks like it's covered with the LNet Health work. I'll take a look at the docs in more detail to see what he had intended.</p> Related LU-10274 Development Rank 1|hzx2yv: Rank (Obsolete) 16881