[LU-12813] sanity 300g,h MDS crash with kernel NULL pointer dereference lod_index_try

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.13.0, Lustre 2.12.2
• Type: Bug
• Priority: Critical
• Reporter: Alexander Boyko
• Assignee: WC Triage
• Resolution: Unresolved
• Labels: mdt

Environment

server 2.12.57_35_g55a7e2d
client 2.11.56

• Severity: 3

Description

sanity 300g 100% reproducible 

== sanity test 300g: check default striped directory for normal directory ============================ 08:40:27 (1569588027)
checking normal_dir 2 1
warning: '--hash-type' and '-t' deprecated, use '--mdt-hash' or '-H' instead
total: 10 open/close in 0.03 seconds: 339.41 ops/second
 - unlinked 0 (time 1569588028 ; total 0 ; last 0)
total: 10 unlinks in 0 seconds: inf unlinks/second
checking normal_dir 1 0
warning: '--hash-type' and '-t' deprecated, use '--mdt-hash' or '-H' instead
total: 10 open/close in 0.02 seconds: 452.16 ops/second
 - unlinked 0 (time 1569588028 ; total 0 ; last 0)
total: 10 unlinks in 0 seconds: inf unlinks/second
checking normal_dir 2 1
warning: '--hash-type' and '-t' deprecated, use '--mdt-hash' or '-H' instead
total: 10 open/close in 0.03 seconds: 395.29 ops/second
 - unlinked 0 (time 1569588028 ; total 0 ; last 0)
total: 10 unlinks in 0 seconds: inf unlinks/second
checking normal_dir 2 -1
warning: '--hash-type' and '-t' deprecated, use '--mdt-hash' or '-H' instead
 

MDS panic with next stack

[144568.438056] Lustre: DEBUG MARKER: == sanity test 300g: check default striped directory for normal directory ============================ 08:40:27 (1569588027)
[144569.662958] BUG: unable to handle kernel NULL pointer dereference at           (null)
[144569.663895] IP: [<ffffffffc1275ad7>] lod_index_try+0x1f7/0x310 [lod]
[144569.664369] PGD 0
[144569.664825] Oops: 0000 [#1] SMP
[144569.665263] Modules linked in: dm_flakey lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) mdc(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) loop jbd2 mbcache joydev nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vboxsf(OE) vboxvideo(OE)
[144569.669497]  snd_intel8x0 snd_ac97_codec ttm iosf_mbi crc32_pclmul ppdev drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ghash_clmulni_intel ac97_bus snd_seq snd_seq_device snd_pcm aesni_intel lrw gf128mul glue_helper snd_timer ablk_helper cryptd snd pcspkr drm vboxguest(OE) i2c_piix4 i2c_core sg soundcore parport_pc parport video nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_net ahci libahci virtio_pci virtio_ring ata_piix crct10dif_pclmul crct10dif_common libata serio_raw crc32c_intel e1000 virtio dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dm_flakey]
[144569.673393] CPU: 1 PID: 11986 Comm: mdt00_003 Tainted: G           OE  ------------   3.10.0-693.21.1.x3.1.9.x86_64 #1
[144569.674516] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[144569.675098] task: ffff880009201fa0 ti: ffff880009230000 task.ti: ffff880009230000
[144569.675663] RIP: 0010:[<ffffffffc1275ad7>]  [<ffffffffc1275ad7>] lod_index_try+0x1f7/0x310 [lod]
[144569.676483] RSP: 0018:ffff880009233950  EFLAGS: 00010202
[144569.677845] RAX: 0000000000000000 RBX: ffff8800595bac80 RCX: 0000000000004000
[144569.678523] RDX: 0000000000000000 RSI: ffff880048b83e00 RDI: ffff88007b0638c0
[144569.679100] RBP: ffff880009233970 R08: ffffffffc114fe8b R09: ffff88000bff8400
[144569.679671] R10: 0000000000000226 R11: 000000000007ffff R12: 0000000000000000
[144569.680275] R13: ffff88007b0638c0 R14: ffffffffc094c120 R15: ffffffffc114fe8b
[144569.680849] FS:  0000000000000000(0000) GS:ffff88007fc80000(0000) knlGS:0000000000000000
[144569.681419] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[144569.682037] CR2: 0000000000000000 CR3: 000000007ae3a000 CR4: 00000000000406e0
[144569.682637] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[144569.683262] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[144569.683871] Call Trace:
[144569.684450]  [<ffffffffc08222ef>] dt_try_as_dir+0x3f/0x50 [obdclass]
[144569.685005]  [<ffffffffc1123bc6>] mdo_declare_index_insert+0x46/0x1c0 [mdd]
[144569.685602]  [<ffffffffc112cfef>] mdd_declare_create+0x69f/0xd50 [mdd]
[144569.686150]  [<ffffffffc11303e2>] mdd_create+0x972/0x1660 [mdd]
[144569.686710]  [<ffffffffc11b1866>] mdt_create+0xb56/0xf50 [mdt]
[144569.687271]  [<ffffffffc0838e31>] ? lprocfs_job_stats_log+0xd1/0x640 [obdclass]
[144569.687855]  [<ffffffffc11b1dcb>] mdt_reint_create+0x16b/0x360 [mdt]
[144569.688417]  [<ffffffffc11b7353>] mdt_reint_rec+0x83/0x210 [mdt]
[144569.688998]  [<ffffffffc1191510>] mdt_reint_internal+0x780/0xb90 [mdt]
[144569.689536]  [<ffffffffc119a2b7>] ? mdt_thread_info_init+0xa7/0x1e0 [mdt]
[144569.690038]  [<ffffffffc119ca27>] mdt_reint+0x67/0x140 [mdt]
[144569.690591]  [<ffffffffc0bc529a>] tgt_request_handle+0x91a/0x15c0 [ptlrpc]
[144569.691086]  [<ffffffffc06fef1e>] ? libcfs_nid2str_r+0xfe/0x130 [lnet]
[144569.691606]  [<ffffffffc0b6a326>] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc]
[144569.692094]  [<ffffffff810c0bd4>] ? __wake_up+0x44/0x50
[144569.692631]  [<ffffffffc0b6de6c>] ptlrpc_main+0xbac/0x1550 [ptlrpc]
[144569.693105]  [<ffffffff810c28a0>] ? finish_task_switch+0x50/0x170
[144569.693607]  [<ffffffffc0b6d2c0>] ? ptlrpc_register_service+0xfa0/0xfa0 [ptlrpc]
[144569.694035]  [<ffffffff810b4031>] kthread+0xd1/0xe0
[144569.694468]  [<ffffffff810b3f60>] ? insert_kthread_work+0x40/0x40
[144569.694901]  [<ffffffff816c1577>] ret_from_fork+0x77/0xb0