https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10113 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/d6ab9a80-af06-11e7-a26c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d6ab9a80-af06-11e7-a26c-5254006e85c2</a>.</p> <p>The sub-test test_111 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Timeout occurred after 653 mins, last suite running was recovery-small, restarting cluster to continue tests </pre> </div></div> <p>in MDS console log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[35694.672927] LustreError: 12891:0:(mgc_request.c:603:do_requeue()) failed processing log: -5 [35694.749290] LustreError: 29168:0:(llog_cat.c:269:llog_cat_id2handle()) lustre-MDT0001-osp-MDT0000: error opening log id [0x1:0x40000405:0x2]:0: rc = -2 [35694.754218] BUG: unable to handle kernel NULL pointer dereference at 0000000000000060 [35694.755113] IP: [&lt;ffffffffc09c504a&gt;] llog_process_thread+0x3a/0x1460 [obdclass] [35694.758457] PGD 0 [35694.758457] Oops: 0000 [#1] SMP [35694.758457] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) loop dm_mod rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr virtio_balloon parport_pc i2c_piix4 parport nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm virtio_blk ata_piix 8139too libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw virtio_pci 8139cp virtio_ring virtio mii i2c_core floppy [last unloaded: libcfs] [35694.758457] CPU: 0 PID: 29168 Comm: lod0000_rec0001 Tainted: G W OE ------------ 3.10.0-693.2.2.el7_lustre.x86_64 #1 [35694.758457] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 [35694.758457] task: ffff88005cae1fa0 ti: ffff880062058000 task.ti: ffff880062058000 [35694.758457] RIP: 0010:[&lt;ffffffffc09c504a&gt;] [&lt;ffffffffc09c504a&gt;] llog_process_thread+0x3a/0x1460 [obdclass] [35694.758457] RSP: 0018:ffff88006205bb28 EFLAGS: 00010202 [35694.758457] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [35694.758457] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88001e5f6a20 [35694.758457] RBP: ffff88006205bbe0 R08: 000000000000000a R09: 000000000000000a [35694.758457] R10: 0000000000000000 R11: 000000000000000f R12: 0000000000000000 [35694.758457] R13: ffff8800561227e0 R14: ffff88001e5f6a20 R15: 0000000000000000 [35694.758457] FS: 0000000000000000(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000 [35694.758457] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [35694.758457] CR2: 0000000000000060 CR3: 00000000019f2000 CR4: 00000000000406f0 [35694.758457] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [35694.758457] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [35694.758457] Stack: [35694.758457] 000000000000000a 0000000000000000 ffff88001da40000 0000000000019be0 [35694.758457] ffff88007d001a00 ffff88006205be58 0000000000000000 ffffffffc0f786b0 [35694.758457] ffff88001e5f6a20 ffff88001e5f6a20 0000000000000000 ffff88006205bbe0 [35694.758457] Call Trace: [35694.758457] [&lt;ffffffffc0f786b0&gt;] ? lodname2mdt_index+0x2f0/0x2f0 [lod] [35694.758457] [&lt;ffffffffc06eeba7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [35694.758457] [&lt;ffffffffc0f786b0&gt;] ? lodname2mdt_index+0x2f0/0x2f0 [lod] [35694.758457] [&lt;ffffffffc09c652c&gt;] llog_process_or_fork+0xbc/0x450 [obdclass] [35694.758457] [&lt;ffffffffc09cba5a&gt;] llog_cat_process_cb+0x20a/0x220 [obdclass] [35694.758457] [&lt;ffffffffc09c5885&gt;] llog_process_thread+0x875/0x1460 [obdclass] [35694.758457] [&lt;ffffffffc09cb850&gt;] ? llog_cat_process_common+0x440/0x440 [obdclass] [35694.758457] [&lt;ffffffffc09c652c&gt;] llog_process_or_fork+0xbc/0x450 [obdclass] [35694.758457] [&lt;ffffffffc09cb850&gt;] ? llog_cat_process_common+0x440/0x440 [obdclass] [35694.758457] [&lt;ffffffffc09ca9d9&gt;] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] [35694.758457] [&lt;ffffffffc0f786b0&gt;] ? lodname2mdt_index+0x2f0/0x2f0 [lod] [35694.758457] [&lt;ffffffffc0f786b0&gt;] ? lodname2mdt_index+0x2f0/0x2f0 [lod] [35694.758457] [&lt;ffffffffc09cab0e&gt;] llog_cat_process+0x2e/0x30 [obdclass] [35694.758457] [&lt;ffffffffc0f74a89&gt;] lod_sub_recovery_thread+0x439/0xc80 [lod] [35694.758457] [&lt;ffffffffc0f74650&gt;] ? lod_trans_stop+0x340/0x340 [lod] [35694.758457] [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 [35694.758457] [&lt;ffffffff810b08c0&gt;] ? insert_kthread_work+0x40/0x40 [35694.758457] [&lt;ffffffff816b4f18&gt;] ret_from_fork+0x58/0x90 [35694.758457] [&lt;ffffffff810b08c0&gt;] ? insert_kthread_work+0x40/0x40 [35694.758457] Code: 41 54 53 48 81 ec 90 00 00 00 4c 8b 27 48 8b 47 18 65 48 8b 34 25 28 00 00 00 48 89 75 d0 31 f6 f6 05 b2 85 d4 ff 01 48 89 7d 88 &lt;4d&gt; 8b 6c 24 60 48 89 45 80 c7 45 c4 00 00 00 00 74 0d f6 05 99 [35694.758457] RIP [&lt;ffffffffc09c504a&gt;] llog_process_thread+0x3a/0x1460 [obdclass] [35694.758457] RSP &lt;ffff88006205bb28&gt; [35694.758457] CR2: 0000000000000060 </pre> </div></div> <p>Info required for matching: recovery-small 111</p> LU-10113

recovery-small test_111: MDS panic after log error

Bug Minor Open Unresolved WC Triage Maloo Thu, 12 Oct 2017 13:18:40 +0000 Tue, 9 Jun 2020 11:50:31 +0000 0 2 <p>Also hit this in:<br/> <a href="https://testing.hpdd.intel.com/test_sets/0e2b0f54-aefd-11e7-9eeb-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0e2b0f54-aefd-11e7-9eeb-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/3c4a530e-a813-11e7-b786-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/3c4a530e-a813-11e7-b786-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/d22ba02e-a595-11e7-bb19-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/d22ba02e-a595-11e7-bb19-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/fc567112-a4d2-11e7-bb19-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/fc567112-a4d2-11e7-bb19-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/476debe8-a443-11e7-bb19-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/476debe8-a443-11e7-bb19-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/093d7572-a232-11e7-b786-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/093d7572-a232-11e7-b786-5254006e85c2</a></p> <p>but it hasn't failed again since 2017-10-12, so it may have been fixed by another patch, or just hiding?</p> <p>Was just hiding. Hit this again on master on 2017-10-20:<br/> <a href="https://testing.hpdd.intel.com/test_sets/35f386e0-b58a-11e7-9d39-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/35f386e0-b58a-11e7-9d39-52540065bddc</a></p> <p>May be fixed by patch <a href="https://review.whamcloud.com/29370/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29370/</a><br/> "<a href="https://jira.whamcloud.com/browse/LU-9994" title="sanity-scrub test_5: test failed to respond and timed out" class="issue-link" data-issue-key="LU-9994"><del>LU-9994</del></a> obdclass: fix llog_cat_id2handle() error handling" landed on 2017-11-03.</p> <p>Let's see if this goes away for patches based after that point.</p> Duplicate LU-10194 Related LU-9994 Development Rank 1|hzzlsv: Rank (Obsolete) 9223372036854775807 Severity