https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1976 Lustre <p>Console <span class="error">&#91;hyperion-rst6&#93;</span> log at 2012-09-18 18:00:00 PDT.<br/> 2012-09-18 18:04:56 Lustre: lustre-MDT0000: haven't heard from client 3beba6a9-a86c-e3b3-e02d-311fe4e1c5ec (at 192.168.118.135@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880288182400, cur 1348016696 expire 1348016546 last 1348016469<br/> 2012-09-18 18:17:21 BUG: unable to handle kernel paging request at 000000008a5e6591<br/> 2012-09-18 18:17:21 IP: <span class="error">&#91;&lt;ffffffffa0855018&gt;&#93;</span> unlock_res_and_lock+0x18/0x40 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-09-18 18:17:21 PGD 0<br/> 2012-09-18 18:17:21 BUG: unable to handle kernel NULL pointer dereference at 0000000000000068<br/> 2012-09-18 18:17:21 IP: <span class="error">&#91;&lt;ffffffff81043b49&gt;&#93;</span> no_context+0x99/0x260</p> <p>MDS fails to dump a stack, but does dump vmcore. at the same time one client dumped vmcore. both dumps are on brent ~cliffw/hyperion/</p> LU-1976

SWL - mds hard crash

Bug Blocker Resolved Fixed nasf Cliff White Tue, 18 Sep 2012 21:29:51 +0000 Mon, 3 Dec 2012 15:04:43 +0000 Sat, 22 Sep 2012 01:13:22 +0000 Lustre 2.3.0 Lustre 2.3.0 Lustre 2.4.0 0 4 <p>Kernel backtrace, MDS:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> bt PID: 4898 TASK: ffff88012df62080 CPU: 7 COMMAND: <span class="code-quote">"mdt03_008"</span> #0 [ffff880121e932b0] machine_kexec at ffffffff8103281b #1 [ffff880121e93310] crash_kexec at ffffffff810ba792 #2 [ffff880121e933e0] oops_end at ffffffff81501700 #3 [ffff880121e93410] die at ffffffff8100f26b #4 [ffff880121e93440] do_trap at ffffffff81500ff4 #5 [ffff880121e934a0] do_invalid_op at ffffffff8100ce35 #6 [ffff880121e93540] invalid_op at ffffffff8100bedb [exception RIP: add_dirent_to_buf+1216] RIP: ffffffffa0db21d0 RSP: ffff880121e935f0 RFLAGS: 00010246 RAX: ffff880181248000 RBX: ffff880268654078 RCX: 00000000000014b5 RDX: ffff880268654098 RSI: 0000000000000046 RDI: ffff88016ff93c00 RBP: ffff880121e936b0 R8: 0000000000000000 R9: 0000000000000000 R10: 0000000000000001 R11: 0000000000000000 R12: ffff880130e16ad0 R13: 0000000000000000 R14: 0000000000000004 R15: 0000000000000020 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #7 [ffff880121e936b8] ldiskfs_add_entry at ffffffffa0db5c5d [ldiskfs] #8 [ffff880121e93758] __osd_ea_add_rec at ffffffffa1001689 [osd_ldiskfs] #9 [ffff880121e937a8] osd_index_ea_insert at ffffffffa100dbeb [osd_ldiskfs] #10 [ffff880121e93838] __mdd_index_insert_only at ffffffffa0eee977 [mdd] #11 [ffff880121e93898] __mdd_index_insert at ffffffffa0eef9b1 [mdd] #12 [ffff880121e938e8] mdd_create at ffffffffa0ef55e3 [mdd] #13 [ffff880121e93a28] cml_create at ffffffffa06a4637 [cmm] #14 [ffff880121e93a78] mdt_reint_open at ffffffffa0f8bb9f [mdt] #15 [ffff880121e93b48] mdt_reint_rec at ffffffffa0f75151 [mdt] #16 [ffff880121e93b68] mdt_reint_internal at ffffffffa0f6e9aa [mdt] #17 [ffff880121e93bb8] mdt_intent_reint at ffffffffa0f6ef7d [mdt] #18 [ffff880121e93c08] mdt_intent_policy at ffffffffa0f6b191 [mdt] #19 [ffff880121e93c48] ldlm_lock_enqueue at ffffffffa0859881 [ptlrpc] #20 [ffff880121e93ca8] ldlm_handle_enqueue0 at ffffffffa08819bf [ptlrpc] #21 [ffff880121e93d18] mdt_enqueue at ffffffffa0f6b506 [mdt] #22 [ffff880121e93d38] mdt_handle_common at ffffffffa0f62802 [mdt] #23 [ffff880121e93d88] mdt_regular_handle at ffffffffa0f636f5 [mdt] #24 [ffff880121e93d98] ptlrpc_server_handle_request at ffffffffa08b199d [ptlrpc] #25 [ffff880121e93e98] ptlrpc_main at ffffffffa08b2f89 [ptlrpc] #26 [ffff880121e93f48] kernel_thread at ffffffff8100c14a </pre> </div></div> <p>Bobijam</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Fanyong</p> <p>Oleg suggested that this might be a good ticket for you to look into</p> <p>Thanks</p> <p>Peter</p> <p>One possible reason may cause such failure: when multiple threads try to recycle empty OI leaves concurrently, the race among those threads may cause some OI index node(s) crazy. Especially when there are some index node(s) split threads in parallel, the cases will become more complex. The crazy OI index node(s) may cause memory crash. That may be why there are some strange failures recently.</p> <p>I am making patch to fix such issue.</p> <p>OI related codes are really bombs <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>This is the patch:</p> <p><a href="http://review.whamcloud.com/#change,4061" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4061</a></p> <p>Landed for 2.3 and 2.4</p> <p>back port to b2_1<br/> <a href="http://review.whamcloud.com/#change,4735" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4735</a></p> Development Rank 1|hzv5hz: Rank (Obsolete) 4425 Severity