[LU-10113] recovery-small test_111: MDS panic after log error

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for Bob Glossman <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/d6ab9a80-af06-11e7-a26c-5254006e85c2.

The sub-test test_111 failed with the following error:

Timeout occurred after 653 mins, last suite running was recovery-small, restarting cluster to continue tests

in MDS console log:

[35694.672927] LustreError: 12891:0:(mgc_request.c:603:do_requeue()) failed processing log: -5
[35694.749290] LustreError: 29168:0:(llog_cat.c:269:llog_cat_id2handle()) lustre-MDT0001-osp-MDT0000: error opening log id [0x1:0x40000405:0x2]:0: rc = -2
[35694.754218] BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
[35694.755113] IP: [<ffffffffc09c504a>] llog_process_thread+0x3a/0x1460 [obdclass]
[35694.758457] PGD 0 
[35694.758457] Oops: 0000 [#1] SMP 
[35694.758457] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) loop dm_mod rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr virtio_balloon parport_pc i2c_piix4 parport nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm virtio_blk ata_piix 8139too libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw virtio_pci 8139cp virtio_ring virtio mii i2c_core floppy [last unloaded: libcfs]
[35694.758457] CPU: 0 PID: 29168 Comm: lod0000_rec0001 Tainted: G        W  OE  ------------   3.10.0-693.2.2.el7_lustre.x86_64 #1
[35694.758457] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[35694.758457] task: ffff88005cae1fa0 ti: ffff880062058000 task.ti: ffff880062058000
[35694.758457] RIP: 0010:[<ffffffffc09c504a>]  [<ffffffffc09c504a>] llog_process_thread+0x3a/0x1460 [obdclass]
[35694.758457] RSP: 0018:ffff88006205bb28  EFLAGS: 00010202
[35694.758457] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[35694.758457] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88001e5f6a20
[35694.758457] RBP: ffff88006205bbe0 R08: 000000000000000a R09: 000000000000000a
[35694.758457] R10: 0000000000000000 R11: 000000000000000f R12: 0000000000000000
[35694.758457] R13: ffff8800561227e0 R14: ffff88001e5f6a20 R15: 0000000000000000
[35694.758457] FS:  0000000000000000(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
[35694.758457] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[35694.758457] CR2: 0000000000000060 CR3: 00000000019f2000 CR4: 00000000000406f0
[35694.758457] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[35694.758457] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[35694.758457] Stack:
[35694.758457]  000000000000000a 0000000000000000 ffff88001da40000 0000000000019be0
[35694.758457]  ffff88007d001a00 ffff88006205be58 0000000000000000 ffffffffc0f786b0
[35694.758457]  ffff88001e5f6a20 ffff88001e5f6a20 0000000000000000 ffff88006205bbe0
[35694.758457] Call Trace:
[35694.758457]  [<ffffffffc0f786b0>] ? lodname2mdt_index+0x2f0/0x2f0 [lod]
[35694.758457]  [<ffffffffc06eeba7>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[35694.758457]  [<ffffffffc0f786b0>] ? lodname2mdt_index+0x2f0/0x2f0 [lod]
[35694.758457]  [<ffffffffc09c652c>] llog_process_or_fork+0xbc/0x450 [obdclass]
[35694.758457]  [<ffffffffc09cba5a>] llog_cat_process_cb+0x20a/0x220 [obdclass]
[35694.758457]  [<ffffffffc09c5885>] llog_process_thread+0x875/0x1460 [obdclass]
[35694.758457]  [<ffffffffc09cb850>] ? llog_cat_process_common+0x440/0x440 [obdclass]
[35694.758457]  [<ffffffffc09c652c>] llog_process_or_fork+0xbc/0x450 [obdclass]
[35694.758457]  [<ffffffffc09cb850>] ? llog_cat_process_common+0x440/0x440 [obdclass]
[35694.758457]  [<ffffffffc09ca9d9>] llog_cat_process_or_fork+0x199/0x2a0 [obdclass]
[35694.758457]  [<ffffffffc0f786b0>] ? lodname2mdt_index+0x2f0/0x2f0 [lod]
[35694.758457]  [<ffffffffc0f786b0>] ? lodname2mdt_index+0x2f0/0x2f0 [lod]
[35694.758457]  [<ffffffffc09cab0e>] llog_cat_process+0x2e/0x30 [obdclass]
[35694.758457]  [<ffffffffc0f74a89>] lod_sub_recovery_thread+0x439/0xc80 [lod]
[35694.758457]  [<ffffffffc0f74650>] ? lod_trans_stop+0x340/0x340 [lod]
[35694.758457]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[35694.758457]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[35694.758457]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[35694.758457]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[35694.758457] Code: 41 54 53 48 81 ec 90 00 00 00 4c 8b 27 48 8b 47 18 65 48 8b 34 25 28 00 00 00 48 89 75 d0 31 f6 f6 05 b2 85 d4 ff 01 48 89 7d 88 <4d> 8b 6c 24 60 48 89 45 80 c7 45 c4 00 00 00 00 74 0d f6 05 99 
[35694.758457] RIP  [<ffffffffc09c504a>] llog_process_thread+0x3a/0x1460 [obdclass]
[35694.758457]  RSP <ffff88006205bb28>
[35694.758457] CR2: 0000000000000060

Info required for matching: recovery-small 111

Issue Links

• is duplicated by: LU-10194 recovery-small test_111: kernel panic on MDS mount
• is related to: LU-9994 sanity-scrub test_5: test failed to respond and timed out