[LU-9725] Mount commands don't return for targets in LFS with DNE and 3 MDTs

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.10.1, Lustre 2.11.0
• Affects Version/s: Lustre 2.10.0
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Tom Nabarro (Inactive)

Description

kernel version: 3.10.0-514.21.1.el7_lustre.x86_64
lustre version: 2.10.0_RC1-1.el7
OS: CentOS Linux release 7.3.1611 (Core)

Failure consistently occurs in test_filesystem_dne.py test_md0_undeleteable() during IML SSI automated test runs testing against lustre b2.10

This is the only test we have which creates a filesystem with 3 MDTs

On recreating LFS (outside of test infrastructure) in a similar configuration with mgs, 3*mdts and 1 ost through IML, all other targets mount commands return successfully but ost mount command never returns.

During when the MDT mount commands are being issued, lots of activity in the kernel messages log including multiple LustreErrors and stack traces, warnings of high cpu usage and then

kernel:NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [lwp_notify_fs1-:13630]

This is on a LDISKF only lfs with DNE enabled. The OST mount command used is as follows and the MDT mount commands are of a similar format:

mount -t lustre /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_disk5 /mnt/fs1-OST0000

The following gists show excerpts from the /var/log/messages log during instances of this type of failure (MDT mounting in DNE):

https://gist.github.com/tanabarr/1adb35a7e7da2581be79df8f45417411
https://gist.github.com/tanabarr/70d3bfa66c4fc474b82c7c02adcda511
https://gist.github.com/tanabarr/9f54584621aacfdeb3899f59687cb918

The last gist link is an extended excerpt giving more contextual log information regarding the attempted mounting of the MDTs and the subsequent CPU load warnings. The entire logfile for that failure instance (in addition to other IML related log files) is attached to this ticket.

original IML ticket: https://github.com/intel-hpdd/intel-manager-for-lustre/issues/108

Issue Links

• is related to: LU-9376 Recovery bug exposed during sanity 103b test
• is related to: LU-8066 Move lustre procfs handling to sysfs and debugfs.

Comments

[Gerrit Updater]

Brian J. Murrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28357
Subject: LU-9725 quota: always deregister lwp
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 728454ee8f85d3074124933d0d83e42f10515500

[Lai Siyao]

Brian, I uploaded a patch, could you help verify it?

[Gerrit Updater]

Lai Siyao ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28356
Subject: LU-9725 quota: always deregister lwp
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 1a5df6bc53e8356d8ae83d4031ea4397ebc03af3

[Brian Murrell (Inactive)]

laisiyao: No problem. Let me know if there is anything else I can do to help.

[Lai Siyao]

Thanks Brian, I'll continue checking the code.

[Brian Murrell (Inactive)]

Brian, is it true 2.10.0 include this fix?

No, 2.10.0 does not include this fix. b2_10 does contain this fix though and that's what we are testing with. You can see from the comment above though that we tested with 2.10.0_5_gbb3c407 which is a build with this commit in it which is 5 commits newer than the landed patch for this ticket.  So we definitely did test the patch from this ticket.

even tag 2.10.50 doesn't include this, can you test with master build?

We cannot test with master due to issues that Jenkins has trying to be an HTTP server. But given the above, we really shouldn't need to test master given that we have tested the patch on b2_10.

[Lai Siyao]

Brian, is it true 2.10.0 include this fix? even tag 2.10.50 doesn't include this, can you test with master build?

[Peter Jones]

Lai

Could you please advise on this one?

Thanks

Peter

[Brian Murrell (Inactive)]

I'm afraid I don't think the patch fixed the problem.

Using:

Lustre: Build Version: 2.10.0_5_gbb3c407

which I believe should have the fix in it I'm still seeing:

[ 1604.696255] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [lwp_notify_test:25841]
[ 1604.703478] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_zfs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache zfs(POE) zunicode(POE) zavl(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate lustre(OE) lmv(OE) mdc(OE) lov(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm sg snd_timer pcspkr virtio_balloon snd i2c_piix4 soundcore parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi virtio_blk virtio_net cirrus drm_kms_helper virtio_scsi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata i2c_core serio_raw floppy virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
[ 1604.785347] CPU: 0 PID: 25841 Comm: lwp_notify_test Tainted: P OE ------------ 3.10.0-514.21.1.el7_lustre.x86_64 #1
[ 1604.798164] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[ 1604.804864] task: ffff88001ac3edd0 ti: ffff88001a204000 task.ti: ffff88001a204000
[ 1604.812330] RIP: 0010:[<ffffffff81327649>] [<ffffffff81327649>] __write_lock_failed+0x9/0x20
[ 1604.820634] RSP: 0018:ffff88001a207e40 EFLAGS: 00000206
[ 1604.828340] RAX: ffff880020eb5400 RBX: ffff88001a207e18 RCX: 0000000000000000
[ 1604.834839] RDX: 0000000000000016 RSI: ffff88001ce8ecc7 RDI: ffff88007a9c1984
[ 1604.842076] RBP: ffff88001a207e40 R08: 0000000000019b20 R09: ffffffffa066f9c1
[ 1604.848577] R10: ffff88007fc19b20 R11: ffffea00006d0400 R12: ffff88001a207dd0
[ 1604.854735] R13: ffff88001ad083c0 R14: ffff88001a207e18 R15: 0000000000000028
[ 1604.861290] FS: 0000000000000000(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
[ 1604.867586] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1604.873392] CR2: 00007fe96aa680e0 CR3: 000000001acfd000 CR4: 00000000000006f0
[ 1604.879558] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 
[ 1604.885757] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 
[ 1604.891532] Stack: 
[ 1604.896912] ffff88001a207e50 ffffffff8168e827 ffff88001a207e70 ffffffffa0ff295a 
[ 1604.903591] ffff88001ce8ec80 ffff880020eb6400 ffff88001a207e98 ffffffffa0689eca 
[ 1604.910593] ffff880020eb6400 ffff880020a03000 ffff880020a030b0 ffff88001a207ec0 
[ 1604.916651] Call Trace: 
[ 1604.922930] [<ffffffff8168e827>] _raw_write_lock+0x17/0x20 
[ 1604.929711] [<ffffffffa0ff295a>] qsd_conn_callback+0x5a/0x160 [lquota] 
[ 1604.935637] [<ffffffffa0689eca>] lustre_notify_lwp_list+0xba/0x100 [obdclass]
[ 1604.941153] [<ffffffffa14d8af6>] lwp_notify_main+0x56/0xc0 [osp]
[ 1604.946248] [<ffffffffa14d8aa0>] ? lwp_import_event+0xb0/0xb0 [osp]
[ 1604.951715] [<ffffffff810b0a4f>] kthread+0xcf/0xe0
[ 1604.956861] [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[ 1604.962093] [<ffffffff81697798>] ret_from_fork+0x58/0x90
[ 1604.967130] [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[ 1604.972471] Code: 66 90 48 89 01 31 c0 66 66 90 c3 b8 f2 ff ff ff 66 66 90 c3 90 90 90 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5 f0 ff 07 f3 90 <83> 3f 01 75 f9 f0 ff 0f 75 f1 5d c3 66 66 2e 0f 1f 84 00 00 00 

 

[Minh Diep]

landed in 2.11

[Brian Murrell (Inactive)]

pjones: Not yet I'm afraid.  We need the stack of 3 packaging patches in combination with this fix to get a functional IML (with Lustre 2.10) system up with which to test the fix for this ticket.  Those three patches look (at least maybe only partially contentiously) in progress so I am hopeful.