https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17379 Lustre <p>The MGC should try all of the MGS NIDs provided on the command line fairly rapidly the <b>first</b> time after mount (without RPC retry) so that the client can detect the case of the MGS running on a backup node quickly. Otherwise, if the mount command-line has many NIDs (4 is very typical, but may be 8 or even 16 in some cases) then the mount can be stuck for several minutes trying to find the MGS running on the right node.</p> <p>The MGC should preferably one NID per node first, then the second NID on each node, etc. This can be done efficiently, assuming the NIDs are provided properly with colon-separated <tt>&lt;MGSNODE&gt;:&lt;MGSNODE&gt;</tt> blocks, and comma-separated "<tt>&lt;MGSNID&gt;,&lt;MGSNID&gt;</tt>" entries within that.</p> <p>However, the NIDs are often not listed correctly with "<tt>:</tt>" separators, so if there are more than 4 MGS NIDs on the command-line, then it would be better to to do a "bisection" of the NIDs to best handle the case of 2/4/8 interfaces per node vs. 2/4/8 separate nodes. For example, try NIDs in order <tt>0, nids/2, nids/4, nids*3/4</tt> for 4 NIDs, then <tt>nids/8, nids*5/8, nids*3/8, nids*7/8</tt> for 8 NIDs, then <tt>nids/16, nids*9/16, nids*5/16, nids*13/16, nids*3/16, nids*11/16, nids*7/16, nids*15/16</tt> for 16 NIDs, and similarly for 32 NIDs (the maximum).</p> <p>It should be fairly quick to determine if the MGS is <b>not</b> responding on a particular NID, because the client will get a rapid error response (e.g. <tt>-ENODEV</tt> or <tt>-ENOTCONN</tt> or <tt>-EHOSTUNREACH</tt> with a short RPC timeout) so in that case it should try all of the NIDs once quickly. If it gets <tt>ETIMEDOUT</tt> that might mean the node is unavailable or overloaded, or it might mean the MGS is not running yet, so the client should back off and retry the NIDs with a longer timeout after the initial burst. </p> <p>However, in most cases the MGS should be running on <em>some</em> node and it just needs to avoid going into slow "backoff" mode until <b>after</b> it has tried all of the NIDs at least once.</p> <p>It would make sense in this case to quiet the "connecting to MGS not running on this node" message for MGS connections so that it doesn't spam the console. </p> LU-17379

try MGS NIDs more quickly at initial mount

Technical task LU-16738 Minor In Progress Unresolved Mikhail Pershin Andreas Dilger Tue, 19 Dec 2023 22:57:44 +0000 Wed, 7 Feb 2024 19:57:05 +0000 Lustre 2.14.0 Lustre 2.15.3 0 8 <p>While investigating this more closely I've found that is not as trivial as it seems. First of all there is time limit for <tt>mgc_enqueue()</tt> set inside function itself:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>   /* Limit how long we will wait for the enqueue to complete */     req-&gt;rq_delay_limit = short_limit ? 5 : MGC_ENQUEUE_LIMIT(exp-&gt;exp_obd);</pre> </div></div> <p>and it is the main reason why only limited amount of NIDs are checked - that is how many attempt were fit in that limit. Note, that limit is just a time enough to find MGS on <b>second</b> node. Basically that is all we can guarantee. I suppose at time that was introduced we were not aware about 4 failover nodes for MGS, not speaking about 16 even.</p> <p>Interesting that with patch from <a href="https://jira.whamcloud.com/browse/LU-17357" title="Client can use incorrect sec flavor when MGT is relocated" class="issue-link" data-issue-key="LU-17357">LU-17357</a> we would always wait for sptlrpc config with timeout enough to scan all mgs nodes despite send limit values above. That means in turn there is no need to set that limit to shorter values as that causes only more often re-queue attempts.</p> <p>So the problem with exiting without scanning all mgs nodes should be resolved by that long enough waiting for stlrpc config and this ticket is more targeted now on reducing total amount of that waiting time</p> <p>First important notice about NIDs scanning - each mgs node separated by ':' is added in connection list of import and can be managed in <tt>import_select_connection()</tt> in any way we would choose, either time-based like now or by bisect proposed by Andreas or by any other way. Meanwhile the mgs NIDs inside single node separated by ',' are out of reach because they are added to connection at LNET level and only LNET is managing them, they are not even visible via <tt>lctl get_param mgc.*.import</tt> unlike connections. That makes non-trivial task to try just first NIDs on each node, then second and so on. Basically that means <tt>import_select_connection()</tt> should be able to notify LNET it needs to try just single NID from many</p> <p>Another way to improve - we can reduce timeout for the first round on connection attempts based ob amount of them in connection list. Now it is always 5s for first attempt. It can be reduced to lesser value if there are many connection linearly or by other means, though it makes no sense to use value less than <tt>obd_get_at_min</tt> probably.</p> <p>As for current working schema of <tt>import_select_connection()</tt> - it used <tt>last_attempt</tt> time per connection to choose the least recently used one when there is no other preference. For any new connection its <tt>last_attempt</tt> value is 0 and such one will be used preferably. That means the first round is linear, it goes from connection list head to its end, gets connection with 0 <tt>last_attempt</tt> and try, then next one in list will be tried as having <tt>0</tt> and so on.</p> <p>So proposed bisect approach (or any other) can be done over those connections still having <tt>0</tt> (not yet tried) while we have any in the list.</p> <p>When all have non-zero last_attempt values then we also have options - to use the least recent (it is supposed to keep initial connection order, not exactly due to 1s time granularity but still), or think about other approaches , e.g. try just <tt>imp_last_success_conn</tt> if exists or maybe it makes sense to remember primary nodes and try them first always on each new round, as HA would try to return nodes there after all</p> <p> </p> <p>Mike, thanks for digging into the details here.</p> <p>It looks like in <tt>import_select_connection()</tt> the client could try to <em>connect</em> to the different NIDs to see which one is alive, rather than waiting on each one separately? That could potentially be done "semi-parallel", like:</p> <ul class="alternate" type="square"> <li>send connect to first NID with 30s or longer RPC timeout (in case server is busy)</li> <li>wait 5s for reply, check if any NID has connected</li> <li>if no connect yet, send to next NID</li> <li>when connect is completed to some NID, set flag in export to indicate no more connections to be tried</li> </ul> <p>We would probably want to quiet the "initial connect" messages from the clients, maybe with <tt>MSG_CONNECT_INITIAL</tt> so that they don't spam the server logs when trying all the servers with "<tt>LustreError: 137-5: lfs00-MDT0005_UUID: not available for connect from 10.89.104.111@tcp (no target)</tt>" when all of the clients are trying to connect to different servers.</p> <p>So far the most troublesome case is when some NIDs in NID list are unavailable, e.g. bad address or missing network:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># time mount -t lustre 192.168.56.11@tcp:192.168.6.12@tcp:192.168.56.13@tcp:192.168.56.14@tcp:192.168.56.101@tcp:/lustre /mnt/lustre real    0m59.289s user    0m0.003s sys    0m0.051s </pre> </div></div> <p>Note the second node is <tt>192.168.6.12</tt> which has no interface on current node. If all addresses are on x.x.56.x network then mount took about 12s to reach correct address which is the last one in the list. So as expected it takes about 4s per node to try.</p> <p>But the situation goes bad if any NID is from missing network.  The request is expired by ptlrpc timeout as expected in about 10s, according with deadline:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00100000:2.0:1706879770.016570:0:6197:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1706879760/real 0]  req@ffff8800a5fb93c0 x1789792650076096/t0(0) o250-&gt;MGC192.168.56.11@tcp@192.168.6.12@tcp:26/25 lens 520/544 e 0 to 1 dl 1706879770 ref 2 fl Rpc:XNr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0</pre> </div></div> <p>but it has no further effect as it starts to wait for LNet to unlink it, which happens after about 40s:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00000040:1.0:1706879811.687504:0:6197:0:(lustre_net.h:2443:ptlrpc_rqphase_move()) @@@ move request phase from UnregRPC to Rpc  req@ffff8800a5fb93c0 x1789792650076096/t0(0) o250-&gt;MGC192.168.56.11@tcp@192.168.6.12@tcp:26/25 lens 520/544 e 0 to 1 dl 1706879770 ref 1 fl UnregRPC:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0</pre> </div></div> <p>And the reason of such behavior is lnet peer discovery, which started when RPC was sent:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00000200:2.0:1706879760.019547:0:6197:0:(niobuf.c:86:ptl_send_buf()) Sending 520 bytes to portal 26, xid 1789792650076096, offset 0 00000400:00000200:2.0:1706879760.019570:0:6197:0:(lib-move.c:5284:LNetPut()) LNetPut -&gt; 12345-192.168.6.12@tcp 00000400:00000200:2.0:1706879760.019605:0:6197:0:(peer.c:2385:lnet_peer_queue_for_discovery()) Queue peer 192.168.6.12@tcp: -114</pre> </div></div> <p>it fails and exit but discovery is going in background. And this RPC may proceed only when discovery is timed out despite all RPC deadlines and timeouts:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000400:00000200:1.0:1706879811.687440:0:6196:0:(peer.c:3061:lnet_discovery_event_handler()) Received event: 6 00000400:00000200:1.0:1706879811.687442:0:6196:0:(peer.c:2385:lnet_peer_queue_for_discovery()) Queue peer 192.168.6.12@tcp: -114 00000400:00000010:1.0:1706879811.687443:0:6196:0:(api-ni.c:1821:lnet_ping_buffer_free()) kfreed 'pbuf': 281 at ffff88011151fba8. 00000400:00000200:1.0:1706879811.687447:0:6196:0:(peer.c:3717:lnet_peer_ping_failed()) peer 192.168.6.12@tcp:-110 00000400:00000200:1.0:1706879811.687449:0:6196:0:(peer.c:4092:lnet_peer_discovery()) peer 192.168.6.12@tcp(ffff8800a5ea06a8) state 0x102060 rc -110 00000400:00000200:1.0:1706879811.687450:0:6196:0:(peer.c:2401:lnet_peer_discovery_complete()) Discovery complete. Dequeue peer 192.168.6.12@tcp</pre> </div></div> <p>So in general we need some way either to prevent peer discovery (and <tt>lnetctl set discovery 0</tt> doesn't work) for particular RPCs at least or to manage discovery timeout to be not longer than RPC deadline. Also, we could consider to force discovery to stop when request is expired, no need to discover anything already, we are just waiting for nothing. </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ssmirnov" class="user-hover" rel="ssmirnov">ssmirnov</a> , can you assist with that and propose possible solutions maybe?</p> <p> </p> <p> </p> <p> </p> <p>Yes, I can see the same even with just two NIDs in the mount command out of which the first NID is unreachable. </p> <p>Default lnet_transaction_timeout of 150 is enough in this case for the mount to fail. Reducing lnet_transaction_timeout to 30 allows the mount to succeed. We're going to be hearing about this from the field I'm sure.</p> <p>I haven't tried a mount with lots of ":"-separated NIDs, but it looks like the supplied NIDs are being discovered in the background in parallel. In my test, discovery for the second (reachable) NID completed almost immediately. However, lustre probably just didn't know it could actually talk to it.</p> <p>Aside from timeout manipulation, it seems that lustre could benefit from knowing that the peer is reachable before firing off a request to it. Not sure what would be the best way to accomplish this within the current architecture. Maybe registering some sort of (optional) callback so that after calling LnetAddPeer lustre gets notified that the peer got added successfully, i.e. discovery is done? If there's a lustre thread waiting on these events and acting on them in the background, this could work. There may be possibly unwanted effects from this: for example, if the first listed server is slower than the second, the second one will be picked for the mount ahead of the first.</p> <p> </p> <p> </p> <p> </p> <p> </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ssmirnov" class="user-hover" rel="ssmirnov">ssmirnov</a> , can we avoid such behavior somehow? The current problem is that Lustre get RPC failed in 10s but can't proceed further until LNET discovery is timed out. That is unexpected behavior, we shouldn't wait about 40s for RPC with 10s deadline. So my question is that possible somehow to notify LNET that we don't need to wait for that particular lnet_libmd? Right now when request is timed out then inside <tt>ptlrpc_unregister_reply()</tt> we call <tt>LNetMDUnlink()</tt> for reply md and for request md if it is not unlinked yet (which means it is not yet sent as in our case), technically that unlink for request md should invoke <tt>request_out_callback()</tt> and finalize RPC processing as I'd expect. But on practice that doesn't work, because MD is referenced:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000400:00000200:0.0:1706967574.984808:0:30492:0:(lib-md.c:64:lnet_md_unlink()) Queueing unlink of md ffff8800934b2f78</pre> </div></div> <p>and that unlink happens only when discovery is done:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000400:00000200:0.0:1706967618.538695:0:30491:0:(peer.c:3061:lnet_discovery_event_handler()) Received event: 6 00000400:00000200:0.0:1706967618.538696:0:30491:0:(peer.c:2385:lnet_peer_queue_for_discovery()) Queue peer 192.168.6.12@tcp: -114 00000400:00000200:0.0:1706967618.538703:0:30491:0:(peer.c:4092:lnet_peer_discovery()) peer 192.168.6.12@tcp(ffff8800a1fc9dd8) state 0x102060 rc -110 00000400:00000200:0.0:1706967618.538704:0:30491:0:(peer.c:2401:lnet_peer_discovery_complete()) Discovery complete. Dequeue peer 192.168.6.12@tcp 00000400:00000200:0.0:1706967618.538706:0:30491:0:(lib-msg.c:1020:lnet_is_health_check()) msg ffff8800ba545448 not committed for send or receive 00000400:00000200:0.0:1706967618.538706:0:30491:0:(lib-md.c:68:lnet_md_unlink()) Unlinking md ffff8800934b2f7</pre> </div></div> <p>and only now RPC is finalized. Is there any way to don't delay md unlink but abort that MD sending immediately? I mean just technically at LNET level.</p> <p> </p> <p>As for idea to don't even try peers which are not yet discovered - can we just check peer status from ptlrpc in some way? Probably we can add new primitive similar to LNetAddPeer() or LNetDebugPeer(), say LNetDiscoverPeer() which would return current discovery status (uptodate, discovering, disabled, etc.) . In that case we could just skip not discovered peers and try first those who discovered and avoid being stuck on dead peers</p> <p>Mike, in some cases the LNet layer is unable to complete the MD and deregister until a timeout finishes, because the RDMA address may have been given out to a remote node. That said, if the host is unreachable or returns an error immediately then that shouldn't happen. </p> <p>Is it possible for the MGC to send separate RPCs asynchronously, so that it doesn't care what happens at the LNet layer? That way, the client can have short timeouts, and try the different MGS NIDs quickly (eg. a few seconds apart), then wait for the first one to successfully reply. We do the same with STATFS and GETATTR RPCs on the OST side. </p> <p>One important point is to silence the console errors on the servers for this case, so the logs are not spammed with "refusing connect from XXXX for MGS" errors (though it should still be printed for other targets since that has been very useful for debugging network issues recently. </p> <p>Andreas, I don't see how that can be done, all RPCs are using import to work with, and import uses only one current connection. That is what <tt>import_select_connection()</tt> does, choose just one from many, then we try to send RPC over it. For your idea we would need to setup imports for each NID to send RPCs to them in parallel, i.e. instead of one MGC import we would need 16 if there are 16 NIDs. And we have to organize them somehow to mark only one as real while others as 'potential'. We could think about how to send RPC with one import but over particular connection maybe, E.g. pings could be sent over each one listed in import and update their status, and import could use that status while choosing a new connection. But that need to be implemented from scratch and it seems that would be somehow the same as LNET peer discovery but at ptlrpc level. Right now I think that peer discovery does in background almost the same as you describe. So I'd try to get peer info from LNET and use it to choose at least alive peers</p> <p>Mikhail what you described is very similar to <a href="https://jira.whamcloud.com/browse/LU-10360" title="use Imperative Recovery logs for client-&gt;MDT/OST connections" class="issue-link" data-issue-key="LU-10360">LU-10360</a> work.</p> <p>Mike, you are right, I wasn't thinking about this side of things. Doing this in parallel at the LNet level would be better. Can the MGC pass all of the peer NIDs to LNet directly to speed up discovery? That would require the MGS NIDs to be specified correctly ("," vs. ":" separators), or possibly have LNet not "trust" the NIDs given on the command-line as all being from the same host. </p> <p>James, I don't see how IR can help with the initial MGS connection, since <a href="https://jira.whamcloud.com/browse/LU-10360" title="use Imperative Recovery logs for client-&gt;MDT/OST connections" class="issue-link" data-issue-key="LU-10360">LU-10360</a> is all about getting the current target NiD(s) <b>from</b> the MGS. That would be a chicken-and-egg problem.</p> <p>LNet does appear to be discovering provided NIDs in parallel, at least in my two-NID ":"-separated test, with the first  of the two NIDs unreachable, the second NID was being discovered immediately. (This may be different with "," separated NIDs.) I don't know what happens at lustre layer exactly, but it looks like it needs to wait for a confirmation on which NID can be used before trying to establish a (lustre-level) connection. That's why I was proposing that there's a thread waiting on "discovery complete" events from LNet. To avoid using provided peers in random order, the thread can wait (for a shorter time) before picking the first available peer ahead of the first listed one. That said, it is definitely possible to add a peer status checker to LNet API if lustre layer prefers polling.</p> <p>Ideally, the MGC code could just give the full list of NIDs to ptlrpc and/or LNet along with the RPC and LNet would sanity the right place. I'm totally fine to change the MGC and/or ptlrpc to do something to notify LNet about all of the MGS NIDs in advance of sending the RPC, so that the right layer can do it the best. In all likelihood, we should probably do the same thing for other connections as well, but they happen in the background and are less noticeable.</p> <p>I think it would be better to have the MGC and other connections just wait for the RPC reply, rather than polling. That depends (AFAIK) on LNet knowing all of the possible NIDs to try for the connection, and I don't <em>think</em> that happens today. It is currently the MGC code that tries all of the NIDs in sequence, and that seems redundant. Also, getting NID handling out of Lustre and into LNet would be a good thing all around. </p> <p>Mike, is this something you could work on? </p> <p>Serguei, can you provide some input on what kind of LNet API you would like to pass multiple NIDs from the MGC down to speed up peer discovery? There may be the risk that the mount command-line contains NIDs that are not correctly specified as belonging to the same host, so there would have to be some defense against that. </p> <p>I was considering a console error to provide feedback to the admin. However, if we get MGS NIDs from round-robin DNS (per <a href="https://jira.whamcloud.com/browse/LU-16738" title="Improve mount.lustre with many MGS NIDs" class="issue-link" data-issue-key="LU-16738">LU-16738</a>) then we wouldn't have any way to distinguish which NIDs belong to the same or different hosts, so if LNet can handle this ambiguity during discovery automatically, then printing a console message is pointless, as would be the need to properly separate the NIDs on the mount command-line, and we could deprecate the use of ":" to separate NIDs and simplify IPv6 address parsing (though using DNS is probably still better than specifying IP addresses directly). </p> <p>I don't think we really need to pass NIDs to LNET, it has all of them added already and is doing discovery in background when they are added and each time when new RPC is sent, so we can rely on current discovery status - note, that upon mount all listed nodes in mount command have just been added to LNET and all are doing discovery already.</p> <p>Right now I think that easier approach would be to keep current scheme when <tt>import_select_connection()</tt> chooses one connection from many based on info about it. So far that is just last_attempt time, we can get peer discovery status (as I see it is just passing status from LNET which is easy to impement), also it could be useful to remember how often connection has been attempted and really used (that would give us stats how often HA uses that node). For me that looks enough to select alive and most used in past node to connect to and that is generic either for MGC or for other imports. </p> <p>Not sure about other details but it looks like that can be done incrementally</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=tappro" class="user-hover" rel="tappro">tappro</a>, I think I can add something like LNetGetPeerStatus which would return current status of the peer provided any of its NIDs. Is that something you could use for starters?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ssmirnov" class="user-hover" rel="ssmirnov">ssmirnov</a> , yes, that would be helpful</p> <p>"Serguei Smirnov &lt;ssmirnov@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53926" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53926</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17379" title="try MGS NIDs more quickly at initial mount" class="issue-link" data-issue-key="LU-17379">LU-17379</a> lnet: add LNetPeerDiscovered to LNet API<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0140bec6cfe4bfd25fbf4088c510867daab3ebb7</p> <p>LNetPeerDiscovered may be useful in the case of ":"-separated NIDs, but not with ","-separated NIDs in the mount string.</p> <p>As far as I can see, the issue with ","-separated NIDs is that LNetPrimaryNID is called just once - it initiates discovery using the first listed NID (primary?) as a target, but doesn't do anything with the knowledge of the non-primary NIDs until the discovery issued to the peer's primary NID fails.</p> <p>I'm going to experiment with modifying LNetPrimaryNID so it may handle this case better, so there may be another patch addressing that.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=tappro" class="user-hover" rel="tappro">tappro</a> </p> <p>Modifying LNetPrimaryNID is going to take a little longer, but this change:</p> <p><a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/53930/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/53930/</a></p> <p>may also be useful in your testing if you use socklnd.</p> <p> </p> <p>"Serguei Smirnov &lt;ssmirnov@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53933" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53933</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17379" title="try MGS NIDs more quickly at initial mount" class="issue-link" data-issue-key="LU-17379">LU-17379</a> lnet: parallelize peer discovery via LNetAddPeer<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a63ead3388b4bcac48f8cf1c9489092af0e92a46</p> <p>"Mikhail Pershin &lt;mpershin@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53937" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53937</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17379" title="try MGS NIDs more quickly at initial mount" class="issue-link" data-issue-key="LU-17379">LU-17379</a> ptlrpc: fix check for callback discard<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: eef6974270086c09c67914257e10732a94db5c9b</p> <p>while testing mount with unavailable NIDs I've found that attempt to call request out callback while unlinking reply doesn't work. The reason is that check for <tt>rq_reply_unlinked</tt> is done too early, that flag is set in reply callback from LNetMDUnlink() which is called after discard check. So I've made that patch above in context of this ticket. It doesn't look as major issue, I don't even think it will have noticeable outcome, but at least it makes original idea to work </p> Related LU-17357 LU-17476 LU-17505 LU-16738 Development Rank 1|i045ef: Rank (Obsolete) 9223372036854775807 Severity