https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6953 Lustre <p>grove-mds1 crashed 2015-07-29 with the following LBUG:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2015-07-29 03:05:17 LustreError: 50126:0:(mdt_handler.c:3409:mdt_recovery()) LBUG 2015-07-29 03:05:17 Call Trace: 2015-07-29 03:05:17 [&lt;ffffffffa07b28f5&gt;] libcfs_debug dumpstack+0x55/0x80 [libcfs] 2015-07-29 03:05:17 Jul 29 03:05:17 [&lt;ffffffffa07b2ef7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 2015-07-29 03:05:17 grove-mds1 kerne [&lt;ffffffffa0fcf9d8&gt;] mdt_handle_common+0x13d8/0x1470 [mdt] 2015-07-29 03:05:17 l: LustreError: [&lt;ffffffffa100b625&gt;] mds_regular_handle+0x15/0x20 [mdt] 2015-07-29 03:05:17 50126:0:(mdt_han [&lt;ffffffffa0b05095&gt;] ptlrpc_server_handle_request+0x305/0xc00 [ptlrpc] 2015-07-29 03:05:17 dler.c:3409:mdt_ [&lt;ffffffffa07b352e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] 2015-07-29 03:05:17 recovery()) LBUG [&lt;ffffffffa07c4845&gt;] ? lc_watchdog_touch+0x65/0x170 [libcfs] </pre> </div></div> <p>It was preceded by a ptlrpc debug message</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2015-07-29 03:05:17 Lustre:50126:0:(mdt_handler.c:4508:mdt_recovery()) @@@ rq_xid 15027...0684 matches last_xid, expected REPLAY or RESENT flag (0) req@ffff...d1400 x15027...0684/t0(0) o101-&gt;28e0...cc83@172.20.15.14@o2ib500:0/0 lens 4616/0 e 0 to 0 dl 1438165072 ref 1 fl Interpret:/0/ffffffff rc 0/-1 </pre> </div></div> <p>For this system, I cannot extract bulk logs and add them to the ticket. We do we have a crash dump and console logs, I can obtain specific information that would help.</p> <p>The mds was under severe memory pressure at the time of the lbug. </p> <p>The MDS was responding very slowly at the time. At 3:05:03 it appears to have dropped 84,316 timed out requests (output from one DEBUG_REQ() call from within ptlrpc_server_handle_request() appears in the console log, followed by Skipped 84315 previous similar messages).</p> lustre-2.5.4-4chaos_2.6.32_504.16.2.1chaos.ch5.3.x86_64.x86_64 LU-6953

LustreError: 50126:0:(mdt_handler.c:3409:mdt_recovery()) LBUG

Bug Major Resolved Cannot Reproduce Mikhail Pershin Olaf Faaland llnl Tue, 4 Aug 2015 16:52:53 +0000 Sat, 10 Oct 2015 14:18:29 +0000 Mon, 5 Oct 2015 22:09:39 +0000 0 5 <p>Mike</p> <p>Could you please advise here?</p> <p>Thanks</p> <p>Peter</p> <p>So this is another case of "client sent us something that we don't understand, let's panic", I guess.<br/> We need to drop this LBUG a tthe very least as the first step and avoid this crash.<br/> Though I am not sure how the condition might arise at all.</p> <p>Also what do you mean you cannot extract bulk logs, crash+ the module to extract the logs is not working?</p> <p>He meant mostly that those things cannot be shared outside of the lab.</p> <p>Olaf, is that possible to find more information about the request with that XID in log? Especially in client log where request was sent from. We have two possibilities here - request flag (RESENT or REPLAY) was dropped somehow or XID was assigned improperly. Client log may help to find out was that resent case or normal request.</p> <p>Mikhail, that XID doesn't appear in the client's console log. The client with NID 172.20.15.14@o2ib500 logged nothing but the expected "lost connection" and "connection restored" messages during the hour leading up to the lbug. Unfortunately we have no additional information from the client.</p> <p>I'm extracting the lustre debug logs from the crash dump and I'll check for that XID and post anything I find.</p> <p>Debug logs from the mds crash dump had nothing obviously of interest. The XID in question did not occur in any message other than the one given above. Rest of the log messages are variations on request timed out, request took too long to process, etc.</p> <p>Were there other occurrences of this issue? There is not enough information to solve it, if it is happening regularly then it is possible to add more debug. </p> <p>Mikhail,</p> <p>This has not occurred again, so go ahead and close it. If it happens again we can reopen it.</p> <p>thanks,<br/> Olaf</p> <p>ok thanks</p> Development Rank 1|hzxjqv: Rank (Obsolete) 9223372036854775807 Severity