https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7536 Lustre <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;1&gt;BUG: unable to handle kernel NULL pointer dereference at (null) &lt;1&gt;IP: [&lt;(null)&gt;] (null) &lt;4&gt;PGD 0 &lt;4&gt;Oops: 0010 [#1] SMP &lt;4&gt;last sysfs file: /sys/devices/system/cpu/online &lt;4&gt;CPU 0 &lt;4&gt;Modules linked in: osp(U) mdd(U) lod(U) mdt(U) lfsck(U) mgs(U) mgc(U) osd_ldiskfs(U) lquota(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ksocklnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic sha256_generic libcfs(U) ldiskfs(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipt_REJECT ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 virtio_balloon virtio_net i2c_piix4 i2c_core ext4 jbd2 mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: nf_defrag_ipv4] &lt;4&gt; &lt;4&gt;Pid: 7661, comm: mdt00_016 Not tainted 2.6.32-431.29.2.el6_lustremaster_master__77 #1 Red Hat KVM &lt;4&gt;RIP: 0010:[&lt;0000000000000000&gt;] [&lt;(null)&gt;] (null) &lt;4&gt;RSP: 0018:ffff880056cefca8 EFLAGS: 00010246 &lt;4&gt;RAX: 0000000000000009 RBX: ffff880056c0b800 RCX: ffff88007c2aea20 &lt;4&gt;RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880056c0b800 &lt;4&gt;RBP: ffff880056cefcc0 R08: 0000000000000009 R09: 0000000000000000 &lt;4&gt;R10: ffff880055063a40 R11: 0000000000000080 R12: 0000000000000000 &lt;4&gt;R13: ffff8800558636c0 R14: 0000000000000000 R15: ffff880055863930 &lt;4&gt;FS: 0000000000000000(0000) GS:ffff880002200000(0000) knlGS:0000000000000000 &lt;4&gt;CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b &lt;4&gt;CR2: 0000000000000000 CR3: 0000000079ad2000 CR4: 00000000000006f0 &lt;4&gt;DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 &lt;4&gt;DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 &lt;4&gt;Process mdt00_016 (pid: 7661, threadinfo ffff880056cee000, task ffff880079413500) &lt;4&gt;Stack: &lt;4&gt; ffffffffa0d9dc85 0000000000000001 ffff880056c0b800 ffff880056cefd00 &lt;4&gt;&lt;d&gt; ffffffffa0d7cc79 ffff880056cefd00 ffff880057193bc0 ffff880056c0b800 &lt;4&gt;&lt;d&gt; 0000000000000009 0000000000000107 ffff880037bfe0b0 ffff880056cefd40 &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffffa0d9dc85&gt;] ? mdt_reconstruct+0x45/0x120 [mdt] &lt;4&gt; [&lt;ffffffffa0d7cc79&gt;] mdt_reint_internal+0xa89/0xb80 [mdt] &lt;4&gt; [&lt;ffffffffa0d7d20b&gt;] mdt_reint+0x6b/0x120 [mdt] &lt;4&gt; [&lt;ffffffffa073d6fc&gt;] tgt_request_handle+0x8bc/0x12e0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa06e4a21&gt;] ptlrpc_main+0xe41/0x1910 [ptlrpc] &lt;4&gt; [&lt;ffffffffa06e3be0&gt;] ? ptlrpc_main+0x0/0x1910 [ptlrpc] &lt;4&gt; [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 &lt;4&gt; [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 &lt;4&gt; [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 &lt;4&gt; [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 &lt;4&gt;Code: Bad RIP value. &lt;1&gt;RIP [&lt;(null)&gt;] (null) &lt;4&gt; RSP &lt;ffff880056cefca8&gt; &lt;4&gt;CR2: 0000000000000000 </pre> </div></div> Server 2.7.63 Client 2.7.63<br/> Configuration : 4 node_dne_singlemds - 1 MDS with DNE, 1 OSS, 2 clients<br/> Release<br/> 2.6.32_431.29.2.el6_lustremaster_master__77_g2d11035 Build Date: Sat 05 Dec 2015 05:40:23 PM UTC<br/> LU-7536

racer_1 test: mds crash : BUG: unable to handle kernel NULL pointer dereference at (null) Oops: 0010 [#1] SMP

Bug Blocker Resolved Fixed WC Triage parinay v kondekar patch Thu, 10 Dec 2015 04:42:11 +0000 Tue, 19 Mar 2019 15:40:27 +0000 Wed, 22 Jun 2016 16:09:08 +0000 Lustre 2.8.0 Lustre 2.9.0 0 11 <p>Looks similar to - <a href="https://jira.hpdd.intel.com/browse/LU-5735" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-5735</a></p> <p>Another instance failing with the same error as above for tag 2.7.66 for FULL - EL6.7 Server/EL6.7 Client - DNE , master build# 3314.<br/> <a href="https://testing.hpdd.intel.com/test_sets/7ae45854-ca83-11e5-9215-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7ae45854-ca83-11e5-9215-5254006e85c2</a></p> <p>Another instance for FULL - EL7.1 Server/EL7.1 Client - DNE, master, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/9e3f6a90-cac5-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9e3f6a90-cac5-11e5-9609-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 - EL6.7 Server/EL6.7 Client - DNE, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/7ae45854-ca83-11e5-9215-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7ae45854-ca83-11e5-9215-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 -EL7.1 Server/EL7.1 Client - DNE, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/9e3f6a90-cac5-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9e3f6a90-cac5-11e5-9609-5254006e85c2</a></p> <p>jadhav.vikram (jadhav.vikram@seagate.com) uploaded a new patch: <a href="http://review.whamcloud.com/20395" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20395</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7536" title="racer_1 test: mds crash : BUG: unable to handle kernel NULL pointer dereference at (null) Oops: 0010 [#1] SMP " class="issue-link" data-issue-key="LU-7536"><del>LU-7536</del></a> mdt: handling NULL dereference in mdt_reconstruct<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a8600c93254140bf8e49f2d4d82b85cf882518f8</p> <p>Looks like re-constructor function correspond to op-code REINT_MIGRATE is not present.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">crash&gt; mdt_thread_info ffff880056c0b800 struct mdt_thread_info { mti_pill = 0xffff880055863930, mti_exp = 0xffff880079f5fc00, .... mti_rr = { rr_opcode = REINT_MIGRATE, rr_handle = 0x0, rr_fid1 = 0xffff88007c84bdd0, rr_fid2 = 0xffff88007c84bde0, rr_name = { ln_name = 0xffff88007c84bf20 <span class="code-quote">"5"</span>, ln_namelen = 1 ..... } crash&gt; REINT_MIGRATE <span class="code-keyword">enum</span> (unknown) = 9 crash&gt; p/x reconstructors $1 = {0x6f6365725f74646d, 0x5f7463757274736e, 0x6d00657461657263, 0x63656a626f5f7464, 0x7475705f74, 0x0, 0x6f6365725f74646d, 0x5f7463757274736e, 0x72747461746573, 0x0} crash&gt; p/x reconstructors[9] $2 = 0x0 crash&gt; typedef <span class="code-keyword">enum</span> { REINT_SETATTR = 1, REINT_CREATE = 2, ...... REINT_MIGRATE = 9, REINT_MAX } mds_reint_t, mdt_reint_t; <span class="code-keyword">static</span> mdt_reconstructor reconstructors[REINT_MAX] = { [REINT_SETATTR] = mdt_reconstruct_setattr, [REINT_CREATE] = mdt_reconstruct_create, [REINT_LINK] = mdt_reconstruct_generic, [REINT_UNLINK] = mdt_reconstruct_generic, [REINT_RENAME] = mdt_reconstruct_generic, [REINT_OPEN] = mdt_reconstruct_open, [REINT_SETXATTR] = mdt_reconstruct_generic }; </pre> </div></div> <p>Got NULL dereference when i purposely hard coded not existing op-code value in mdt_reconstruct()</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">mti-&gt;mti_rr.rr_opcode = 9; reconstructors[mti-&gt;mti_rr.rr_opcode](mti, lhc); </pre> </div></div> <p>As per given analysis given in above comment mdt_reconstruct function corresponding to op-code REINT_MIGRATE is not present, so adding check for non defined re-constructor function for given opcode. The patch <a href="http://review.whamcloud.com/#/c/20395/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/20395/</a> created to fix the same.<br/> I have verified the fix by adding hard coded op-codes.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20395/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20395/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7536" title="racer_1 test: mds crash : BUG: unable to handle kernel NULL pointer dereference at (null) Oops: 0010 [#1] SMP " class="issue-link" data-issue-key="LU-7536"><del>LU-7536</del></a> mdt: handling NULL dereference in mdt_reconstruct<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: aa40787eee1835e4f84a40caa96fb232354bd799</p> <p>Patch has landed to master for 2.9.0</p> Blocker LU-5735 Duplicate Related Development Rank 1|hzxvfj: Rank (Obsolete) 9223372036854775807 Severity