https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1717 Lustre <p>Running SWL tests, mix of various (IOR, mdtest, simul, mir, fdtree) <br/> Seeing this sequence repeatedly, lustre dump sent to ftp site. File Name: lu-1442.dump.gz</p> <p>Aug 7 09:51:29 ehyperion-rst6 kernel: LustreError: 29701:0:(mdt_recovery.c:611:mdt_steal_ack_locks()) Resent req xid 1409327760945603 has mismatched opc: new 101 old 0<br/> Aug 7 09:51:29 ehyperion-rst6 kernel: LustreError: 29701:0:(mdt_recovery.c:611:mdt_steal_ack_locks()) Skipped 5 previous similar messages<br/> Aug 7 09:51:29 ehyperion-rst6 kernel: Lustre: 29701:0:(mdt_recovery.c:622:mdt_steal_ack_locks()) Stealing 1 locks from rs ffff8802d1a96000 x1409327760945603.t537972417766 o0 NID 192.168.117.9@o2ib1<br/> Aug 7 09:51:29 ehyperion-rst6 kernel: Lustre: 29701:0:(mdt_recovery.c:622:mdt_steal_ack_locks()) Skipped 5 previous similar messages<br/> Aug 7 09:51:29 ehyperion-rst6 kernel: Lustre: 4710:0:(service.c:2095:ptlrpc_handle_rs()) All locks stolen from rs ffff8802d1a96000 x1409327760945603.t537972417766 o0 NID 192.168.117.9@o2ib1</p> LLNL Hyperion, CHAOS 5 servers/clients, Lustre 2.2.92 LU-1717

mdt_recovery.c:611:mdt_steal_ack_locks()) Resent req xid XXX has mismatched opc: new 101 old 0

Bug Minor Resolved Fixed Li Wei Cliff White topsequoia Tue, 7 Aug 2012 13:05:50 +0000 Thu, 12 Jun 2014 02:42:24 +0000 Mon, 15 Oct 2012 22:10:25 +0000 Lustre 2.3.0 Lustre 2.4.0 0 6 <p>I think this might be a case of improper init:</p> <p>in target_send_reply() we have:</p> <p> rs-&gt;rs_opc = lustre_msg_get_opc(rs-&gt;rs_msg);</p> <p>rs-&gt;rs_msg does not seem to be initialized in a proper way yet (points to yet uninitialized reply buffer).</p> <p>so probably should be lustre_msg_get_opc(req-&gt;rq_reqmsg);</p> <p>The Stealing..../Stolen... messages should be silenced.</p> <p>Technically these messages should only be seen when there is a lost reply enroute to client</p> <p>We are also seeing this repeatedly.</p> <p>In particular, we are still seeing this in newer master code at <a href="https://github.com/chaos/lustre/tree/2.3.53-3chaos" class="external-link" target="_blank" rel="nofollow noopener">2.3.53-3chaos</a>.</p> <p>We are seening this repeatedly on our 2.1.2 and 2.1.3 servers, but I can not pin this to a particular reported problem.</p> <p>The particular problem you'd see this in is when a reply from server to client was lost and client did a resend.<br/> The message is harmless (and wrong, and will be fixed).</p> <p>Oleg, if your assumption about lost replies is correct, then I think we have a bigger problem here. We do not have lnet routers on Sequoia so we should have a reliable communication fabric.</p> <p>How are we losing messages so often??</p> <p>I don't really know how are you losing the messages.</p> <p>The resent xid could only occur if a reply was not seen by a client and it decided to resend the message (there probably should be a client-side message about that too).<br/> The specific message you see could only happen when that lost reply happened to be one for a so called "difficult" reply - where a lock is being returned to the client.</p> <p><a href="http://review.whamcloud.com/4271" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4271</a></p> <p>This does what Oleg suggested.</p> <p>Patch landed</p> Duplicate Related LU-2187 Development Rank 1|hzvgpb: Rank (Obsolete) 6355 Severity