Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-10105] kernel:Kernel panic - not syncing: LBUG https://jira.whamcloud.com/browse/LU-10105 Lustre <p>When attempt to create a *new *file from lustre client node, it's throwing the following panic error and rebooting the lustre client node. After the reboot, the file that was tried to create is present with 0 bytes and can be used to write data into it. Creation of new directories are fine. The problem is with creation of new files. Existing files can be updated but creation of new files from lustre client node is throwing the following panic error and rebooting the lustre client node.</p> <p>Message from syslogd@10-64-7-142 at Oct 9 16:19:18 ...<br/> kernel:LustreError: 4337:0:(dcache.c:188:ll_d_init()) ASSERTION( de-&gt;d_op == &amp;ll_d_ops ) failed:</p> <p>Message from syslogd@10-64-7-142 at Oct 9 16:19:18 ...<br/> kernel:LustreError: 4337:0:(dcache.c:188:ll_d_init()) LBUG</p> <p>Message from syslogd@10-64-7-142 at Oct 9 16:19:19 ...<br/> kernel:Kernel panic - not syncing: LBUG</p> RHEL 7.4 plus 3rd party kernel modules LU-10105 kernel:Kernel panic - not syncing: LBUG Bug Blocker Resolved Won't Do Brad Hoagland Shankar Vellaichamy Mon, 9 Oct 2017 15:38:40 +0000 Mon, 25 May 2020 14:19:40 +0000 Thu, 19 Apr 2018 16:19:24 +0000 Lustre 2.10.1 0 6 <p>Message from syslogd@10-64-7-142 at Oct 9 16:51:16 ...<br/> kernel:LustreError: 7760:0:(dcache.c:188:ll_d_init()) ASSERTION( de-&gt;d_op == &amp;ll_d_ops ) failed:</p> <p>Message from syslogd@10-64-7-142 at Oct 9 16:51:16 ...<br/> kernel:LustreError: 7760:0:(dcache.c:188:ll_d_init()) LBUG<br/> Oct 9 16:51:16 10-64-7-142 kernel: LustreError: 7760:0:(dcache.c:188:ll_d_init()) ASSERTION( de-&gt;d_op == &amp;ll_d_ops ) failed:<br/> Oct 9 16:51:16 10-64-7-142 kernel: LustreError: 7760:0:(dcache.c:188:ll_d_init()) LBUG<br/> Oct 9 16:51:16 10-64-7-142 kernel: Pid: 7760, comm: touch<br/> Oct 9 16:51:16 10-64-7-142 kernel: #012Call Trace:<br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffffc06957ae&gt;&#93;</span> libcfs_call_trace+0x4e/0x60 <span class="error">&#91;libcfs&#93;</span><br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffffc069583c&gt;&#93;</span> lbug_with_loc+0x4c/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffffc0ba0d2e&gt;&#93;</span> ll_d_init+0x2de/0x420 <span class="error">&#91;lustre&#93;</span><br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffffc09df53d&gt;&#93;</span> ? __req_capsule_get+0x15d/0x700 <span class="error">&#91;ptlrpc&#93;</span><br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffffc0be3ba8&gt;&#93;</span> ll_splice_alias+0x1b8/0x320 <span class="error">&#91;lustre&#93;</span><br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffffc0be3d93&gt;&#93;</span> ll_lookup_it_finish+0x83/0x1090 <span class="error">&#91;lustre&#93;</span><br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffffc0b81787&gt;&#93;</span> ? lmv_intent_lock+0xe37/0x1b50 <span class="error">&#91;lmv&#93;</span><br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffff812cb380&gt;&#93;</span> ? security_sid_to_context+0x10/0x20<br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffff812bb875&gt;&#93;</span> ? selinux_dentry_init_security+0xa5/0x110<br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffffc0be2e80&gt;&#93;</span> ? ll_md_blocking_ast+0x0/0x730 <span class="error">&#91;lustre&#93;</span><br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffffc0be563e&gt;&#93;</span> ll_lookup_it+0x89e/0xee0 <span class="error">&#91;lustre&#93;</span><br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffff812ca70d&gt;&#93;</span> ? context_struct_compute_av+0x34d/0x470<br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffffc0be5db7&gt;&#93;</span> ll_atomic_open+0x137/0x12d0 <span class="error">&#91;lustre&#93;</span><br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffff816a318f&gt;&#93;</span> ? avc_compute_av+0x1a3/0x1b5<br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffffc0407138&gt;&#93;</span> rfs_atomic_open+0x148/0x380 <span class="error">&#91;redirfs&#93;</span><br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffff812100fd&gt;&#93;</span> do_last+0xa4d/0x12c0<br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffff81210a32&gt;&#93;</span> path_openat+0xc2/0x490<br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffff8118295b&gt;&#93;</span> ? unlock_page+0x2b/0x30<br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffff81212fcb&gt;&#93;</span> do_filp_open+0x4b/0xb0<br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffff8111ea0c&gt;&#93;</span> ? audit_alloc_name+0x9c/0x160<br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffff8111f6fd&gt;&#93;</span> ? __audit_getname+0x3d/0xb0<br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffff81220249&gt;&#93;</span> ? __alloc_fd+0xa9/0x130<br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffff811ffc13&gt;&#93;</span> do_sys_open+0xf3/0x1f0<br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffff811ffd2e&gt;&#93;</span> SyS_open+0x1e/0x20<br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffffc0421c34&gt;&#93;</span> gsch_open_hook_fn+0x124/0x140 <span class="error">&#91;gsch&#93;</span><br/> Oct 9 16:51:16 10-64-7-142 kernel: <span class="error">&#91;&lt;ffffffff816b5009&gt;&#93;</span> system_call_fastpath+0x16/0x1b</p> <p>Message from syslogd@10-64-7-142 at Oct 9 16:51:16 ...<br/> kernel:Kernel panic - not syncing: LBUG<br/> Oct 9 16:51:16 10-64-7-142 kernel:<br/> Oct 9 16:51:16 10-64-7-142 kernel: Kernel panic - not syncing: LBUG</p> <p>Could you describe the client configuration used here? Especially the how gsch and redirfs are used?</p> <p>Hi John, After stopping ds_agent using /etc/init.d/ds_agent stop, new file creations are successful. Please refer attached ds_agent diagnostic package from output of /opt/ds_agent/dsa_control -d. Could you please support to make the creation of new files work from lustre client with ds-agent active and running.</p> <p><span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/28426/28426_ds-agent-diagnostic.zip" title="ds-agent-diagnostic.zip attached to LU-10105">ds-agent-diagnostic.zip<sup><img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/></sup></a></span></p> <p>Hi Shankar,</p> <p>Do you have a support agreement for this system with Intel?</p> <p>Regards,</p> <p>Brad</p> <p>Hi Brad,  We have just started with Lustre and don't have support agreement for this system with Intel.</p> <p>For the record, I've faced the same crashes which ended up being cause by the same software. <tt>ds_agent</tt> is <tt>Trend Micro Deep Security Agent</tt>. This is part of Trend Deep Security composed of a file system wrapping real filesystem accesses: <tt>redirfs</tt> and a hooking modules <tt>gsch</tt> . Here is the LBUG stack trace:</p> <p> </p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> crash&gt; bt PID: 19992 TASK: ffff9df833f73150 CPU: 1 COMMAND: <span class="code-quote">"touch"</span> #0 [ffff9df7395b3728] machine_kexec at ffffffffa6466044 #1 [ffff9df7395b3788] __crash_kexec at ffffffffa6522ee2 #2 [ffff9df7395b3858] panic at ffffffffa6b7952c #3 [ffff9df7395b38d8] lbug_with_loc at ffffffffc03028cb [libcfs] #4 [ffff9df7395b38f8] ll_d_init at ffffffffc07b9d2e [lustre] #5 [ffff9df7395b3938] ll_splice_alias at ffffffffc07fdab8 [lustre] #6 [ffff9df7395b3980] ll_lookup_it_finish at ffffffffc07fdca3 [lustre] #7 [ffff9df7395b3a48] ll_lookup_it at ffffffffc07ff576 [lustre] #8 [ffff9df7395b3b08] ll_atomic_open at ffffffffc07ffcf7 [lustre] #9 [ffff9df7395b3bc8] rfs_atomic_open at ffffffffc075622b [redirfs] #10 [ffff9df7395b3c70] do_last at ffffffffa665d803 #11 [ffff9df7395b3d20] path_openat at ffffffffa665e1bd #12 [ffff9df7395b3db8] do_filp_open at ffffffffa666040d #13 [ffff9df7395b3e90] do_sys_open at ffffffffa664bfe4 #14 [ffff9df7395b3ef0] sys_open at ffffffffa664c0fe #15 [ffff9df7395b3f00] gsch_open_hook_fn at ffffffffc088a4fa [gsch] #16 [ffff9df7395b3f50] system_call_fastpath at ffffffffa6b92ed2 RIP: 00007fb6a78d3760 RSP: 00007ffee9c8e958 RFLAGS: 00010202 RAX: 0000000000000002 RBX: 00007ffee9c8ecf8 RCX: 0000000000000037 RDX: 00000000000001b6 RSI: 0000000000000941 RDI: 00007ffee9c9056f RBP: 0000000000000000 R8: 0000000000000000 R9: 0000000000000000 R10: 00007ffee9c8dea0 R11: 0000000000000246 R12: 00007ffee9c9056f R13: 00007fb6a7bab2a0 R14: 0000000000000001 R15: 0000000000000000 ORIG_RAX: 0000000000000002 CS: 0033 SS: 002b </pre> </div></div> <p>(See '<tt>gsch</tt>' at the beginning of the call stack)</p> <p>This hook or <tt>redirfs</tt> is replacing the <tt>struct dentry_operations</tt> by its own, where some callbacks are replaced by its own callbacks, making the LASSERT fail.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> crash&gt; p &amp;ll_d_ops $7 = (<span class="code-keyword">const</span> struct dentry_operations *) 0xffffffffc0824200 &lt;ll_d_ops&gt; ... crash&gt; struct dentry_operations 0xffff9df83256e040 struct dentry_operations { d_revalidate = 0xffffffffc07538e0, d_weak_revalidate = 0x0, d_hash = 0x0, d_compare = 0xffffffffc07b9770 &lt;ll_dcompare&gt;, d_delete = 0xffffffffc07b94e0 &lt;ll_ddelete&gt;, d_release = 0xffffffffc0753b30, d_prune = 0x0, d_iput = 0xffffffffc07536a0, d_dname = 0x0, d_automount = 0x0, { d_manage = 0x0, __UNIQUE_ID_rh_kabi_hide17 = { d_manage = 0x0 }, {&lt;No data fields&gt;} } } ffffffffc07538e0 (t) rfs_d_revalidate [redirfs] ffffffffc0753b30 (t) rfs_d_release [redirfs] ffffffffc07536a0 (t) rfs_d_iput [redirfs] </pre> </div></div> <p> </p> <p>This kind of low level hooking is definitely a bad design in my opinion, but I was wondering if this <tt>LASSERT</tt> is really useful and if removing it could be enough to make this software works.</p> <p> </p> Development Rank 1|hzzlkf: Rank (Obsolete) 9223372036854775807 Severity