https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1702 Lustre <p>Message from syslogd@metal at Aug 109:46:26 ...<br/> kernel: LustreError: 3218:0: (mdt_open.c:1035:mdt_reconstruct_open()) AssERTION( (!(rc &lt; 0) I I (lustre_msg_get_transno(req-&gt;r~repmsg) == 0)) ) failed:</p> <p>Message from syslogd@metal at Aug 1 09:46:26 .,.<br/> kernel:LustreError: 3218:0:(mdt_open.c:1035:mdt_reconstruct_open()) LBUG</p> <p>Message from syslogd@metal at Aug 1 09:46:26<br/> kernel:Kernel panic -not syncing: LBUG@o2ib:0/0 lens 192/0 e 0 to 0 dl 1343828771 ref 1 fl Interpret:H/0/ffffffff rc 0/</p> <p>1 Aug 1 09:46:26 metal kernel: LustreError:3218:0:(mdt_open.c:1035:mdt_reconstruct_open()) ASSERTION ( (!(rc &lt; 0) I I (lustre_msg_get_transno(req-&gt;r~repmsg) == 0)) ) failed:</p> <p>Message from syslogd@metal at Aug 1 09:46:26 ...<br/> kernel: LustreError: 3218:0:(mdt_open.c:1035:mdt_reconstruct_open()) AssERTION( (!(rc &lt; 0) I I (lustre_msg_get_transno(req-&gt;r~repmsg) == 0)) ) failed:</p> <p>Message from syslogd@metal at Aug 1 09:46:26 ...<br/> kernel: LustreError: 3218:0:(mdt_open.c:1035:mdt_reconstruct_open()) LBUG</p> <p>Aug 1 09:46:26 metal kernel: LustreError: 3218:0:(mdt_open.c:1035:mdt_reconstruct_open()) LBUG<br/> Aug 1 09:46:26 metal kernel: Pid: 3218, comm: mdt_24<br/> Aug 1 09:46:26 metal kernel:<br/> Aug 1 09:46:26 metal kernel: Call Trace:<br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0422835&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0422d67&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0d6e60b&gt;&#93;</span> mdt_reconstruct_open+0x63b/0x8c0 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0d620fb&gt;&#93;</span> mdt_reconstruct+0x4b/0xb0 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0d511f9&gt;&#93;</span> mdt_reint_internal+0x609/0x7b0 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0d5l5d5&gt;&#93;</span> mdt_intent_reint+0x185/0x4a0 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0d5036l&gt;&#93;</span> mdt_intent_policy+0x2dl/0x600 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0668e39&gt;&#93;</span> ldlm_lock_enqueue+0x2f9/0x830 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa068gef0&gt;&#93;</span> ldlm_handle_enqueue0+0x420/0xd90 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0d506d6&gt;&#93;</span> mdt_enqueue+0x46/0x130 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0d47b9d&gt;&#93;</span> mdt_handle_common+0x74d/0x1400 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0d48925&gt;&#93;</span> mdt_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa06b10l1&gt;&#93;</span> ptlrpc_server_handle_request+0x3cl/0xcb0<span class="error">&#91;ptlrpc&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa04233ee&gt;&#93;</span> ?cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa042de19&gt;&#93;</span> ?lc_watchdog_touch+0x79/0xl10 <span class="error">&#91;libcfs&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa06ab0e2&gt;&#93;</span> ?ptlrpc_wait_event+0xb2/0x2c0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffff8l0519c3&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa06b201f&gt;&#93;</span>ptlrpc_main+0x7lf/0x12l0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa06b1900&gt;&#93;</span> ?ptlrpc_main+0x0/0x12l0 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffff8100c14a&gt;&#93;</span> child_rip+0xa/0x20<br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa06b1900&gt;&#93;</span> ?ptlrpc_main+0x0/0x1210 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa06b1900&gt;&#93;</span> ?ptlrpc_main+0x0/0x1210 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffff8100c140&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Aug 1 09:46:26 metal kernel:<br/> Aug 1 09:46:26 metal kernel: Kernel panic not syncing: LBUG</p> <p>Message from syslogd@meta1 at Aug 109:46:26 ... kernel:Kernel panic -not syncing: LBUG<br/> Aug 1 09:46:26 metal kernel: Pid: 3218, comm: mdt_24 Not tainted 2.6.32-220.4.2.e16.x86_64 #1<br/> Aug 1 09:46:26 metal kernel: Call Trace:<br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffff814ec61a&gt;&#93;</span> ? panic+0x78/0x143<br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0422dbb&gt;&#93;</span> ?lbug_with_loc+0x9b/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0d6e60b&gt;&#93;</span> ?mdt_reconstruct_open+0x63b/0x8c0 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0d620fb&gt;&#93;</span> ?mdt_reconstruct+0x4b/0xb0 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0dS1lf9&gt;&#93;</span> ?mdt_reint_internal+0x609/0x7b0 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0dSlSdS&gt;&#93;</span> ?mdt_intent_reint+0x18S/0x4a0 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0d5036l&gt;&#93;</span> ?mdt_intent~olicy+0x2d1/0x600 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0668e39&gt;&#93;</span> ?ldlm_lock_enqueue+0x2f9/0x830 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa068gef0&gt;&#93;</span> ?ldlm_handle_enqueue0+0x420/0xd90 <span class="error">&#91;ptlrpc&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0dS06d6&gt;&#93;</span> ?mdt_enqueue+0x46/0x130 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0d47b9d&gt;&#93;</span> ?mdt_handle_common+0x74d/0x1400 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa0d48925&gt;&#93;</span> ?mdt_regular_handle+0xlS/0x20 <span class="error">&#91;mdt&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa06b10ll&gt;&#93;</span> ?ptlrpc_server_handle_request+0x3c1/0xcb0<span class="error">&#91;ptlrpc&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa04233ee&gt;&#93;</span> ?cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa042del9&gt;&#93;</span> ?lc_watchdog_touch+0x79/0xl10 <span class="error">&#91;libcfs&#93;</span><br/> Aug 1 09:46:26 metal kernel: <span class="error">&#91;&lt;ffffffffa06ab0e2&gt;&#93;</span> ?ptlrpc_wait_event+0xb2/0x2c0 <span class="error">&#91;ptlrpc&#93;</span></p> Redhat 6.2 x86_64 with infiniband LU-1702

LustreError: 3218:0: (mdt_open.c:1035:mdt_reconstruct_open()) LBUG

Bug Major Resolved Cannot Reproduce Bob Glossman Michael Di Domenico Thu, 2 Aug 2012 15:39:40 +0000 Mon, 29 May 2017 04:53:31 +0000 Mon, 29 May 2017 04:53:31 +0000 Lustre 2.2.0 0 9 <p>Bob will look into this one</p> <p>Michael,<br/> Would really like to know a little more about this problem. Do you have a reproducer? Was it a one-off sort of thing or do you keep seeing it a lot? Are there some syslogs, lustre debug logs, or dmesg logs from around the time of the LBUG that you could provide?</p> <p>The bug seems to be related to load on the machine (which is caused by) heavy scanning of the filesystem. I don't have an exact reproducer, but the machine has crashed several times during some heavy IO periods.</p> <p>I'm not able to pull syslog/dmesg entries in-mass from the system. If there's something specific you're looking for i can search around. i do have lustre logs in /tmp, but because they're binary, i am not able to remove them from the system. if you have commands that will let me instrument the data you need from the log and print it, i can scan it back in and send it over</p> <ul class="alternate" type="square"> <li>i do have lustre logs in /tmp, but because they're binary, i am not able to remove them from the system. -</li> </ul> <p>If the only thing stopping you from sending us lustre logs is that they are binary, you can convert them to human readable text with 'lctl df'. Would that permit you to send them?</p> <p>Yes, converting them to ascii definitely helps, however, the excerpt of lines during the kernel panic is 155k, I'll need to pair that down before i can pull the data from the system. can you tell me if there are recurring lines that i can grep out, i'll ask for the whole file, but i suspect it'll get declined since the result is currently 37MB</p> <p>Hi,</p> <p>We've experienced this as well and believe it to be more related to recovery issues and less general load related.</p> <p>I'll try and get some more data together and post it here so this ticket is more complete.</p> <p>-cf</p> <p>I have seen this on 2.2</p> <p>Close old ticket.</p> Development Rank 1|hzv3g7: Rank (Obsolete) 4058 Severity