[LU-10768] soft lockup on the MDS

Mon, 5 Mar 2018 08:56:31 +0000

https://testing.hpdd.intel.com/test_sessions/01875d8a-a6f9-4b44-8643-5b4c90e05027
https://testing.hpdd.intel.com/test_logs/3049722c-1fb1-11e8-a6ca-52540065bddc/show_text

[13034.022231] Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=trevis-16vm12@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=zfs --device-size=100000 --reformat lustre-ost1/ost1 /dev/lvm-Role_OSS/P1
[13101.161575] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [zpool:15753]
[13101.162559] Modules linked in: osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) zfs(POE) zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) libcfs(OE) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr virtio_balloon i2c_piix4 parport_pc parport nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix virtio_blk 8139too libata crct10dif_pclmul crct10dif_common crc32c_intel 8139cp serio_raw mii virtio_pci i2c_core virtio_ring virtio floppy
[13101.162559] CPU: 0 PID: 15753 Comm: zpool Tainted: P OE ------------ 3.10.0-693.17.1.el7_lustre.x86_64 #1
[13101.162559] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[13101.162559] task: ffff88005809cf10 ti: ffff880079a94000 task.ti: ffff880079a94000
[13101.162559] RIP: 0010:[<ffffffff811f4ea0>] [<ffffffff811f4ea0>] __mem_cgroup_commit_charge+0x30/0x2f0
[13101.162559] RSP: 0018:ffff880079a979a8 EFLAGS: 00010286
[13101.162559] RAX: ffff88007c9690a0 RBX: ffff8800000112d0 RCX: 0000000000000000
[13101.162559] RDX: ffff88007ff850e0 RSI: ffffea0000fa4280 RDI: 000000000003e90a
[13101.162559] RBP: ffff880079a979f0 R08: 0000000000000000 R09: ffff880079543408
[13101.162559] R10: 0000000000000000 R11: ffff880079543408 R12: ffffffff816b9bff
[13101.162559] R13: ffffffff816b9c06 R14: ffffffff816b9c0d R15: ffffffff816b9c14
[13101.162559] FS: 00007efca99a77c0(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
[13101.162559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[13101.162559] CR2: 00007efca9967000 CR3: 0000000056fa0000 CR4: 00000000000606f0
[13101.162559] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[13101.162559] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[13101.162559] Call Trace:
[13101.162559] [<ffffffff811f67f7>] mem_cgroup_charge_common+0x77/0xc0
[13101.162559] [<ffffffff811f885a>] mem_cgroup_cache_charge+0x8a/0xb0
[13101.162559] [<ffffffff81184ef2>] __add_to_page_cache_locked+0x52/0x2a0
[13101.162559] [<ffffffff81185197>] add_to_page_cache_lru+0x37/0xb0
[13101.162559] [<ffffffff81245385>] mpage_readpages+0xb5/0x150
[13101.162559] [<ffffffff8123edf0>] ? I_BDEV+0x10/0x10
[13101.162559] [<ffffffff8123edf0>] ? I_BDEV+0x10/0x10
[13101.162559] [<ffffffff8123f7fd>] blkdev_readpages+0x1d/0x20
[13101.162559] [<ffffffff811912dc>] __do_page_cache_readahead+0x1cc/0x250
[13101.162559] [<ffffffff81191859>] force_page_cache_readahead+0x99/0xe0
[13101.162559] [<ffffffff81191937>] page_cache_sync_readahead+0x97/0xb0
[13101.162559] [<ffffffff81185f6b>] generic_file_aio_read+0x29b/0x790
[13101.162559] [<ffffffff8123fc3c>] blkdev_aio_read+0x4c/0x70
[13101.162559] [<ffffffff8120215d>] do_sync_read+0x8d/0xd0
[13101.162559] [<ffffffff81202b5c>] vfs_read+0x9c/0x170
[13101.162559] [<ffffffff81203a1f>] SyS_read+0x7f/0xe0
[13101.162559] [<ffffffff816b8929>] ? system_call_after_swapgs+0x156/0x214
[13101.162559] [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
[13101.162559] [<ffffffff816b889d>] ? system_call_after_swapgs+0xca/0x214
[13101.162559] Code: 55 48 89 e5 41 57 45 89 c7 41 56 41 55 41 54 49 89 fc 48 89 f7 53 48 89 f3 48 83 ec 20 89 55 d4 89 4d d0 e8 33 41 00 00 49 89 c6 <f0> 41 0f ba 2e 00 19 c0 85 c0 0f 85 8d 02 00 00 31 c0 45 84 ff

Whamcloud Community JIRA

[LU-10768] soft lockup on the MDS