[LU-9854] Lustre 2.10.0 mmap() issues

• Type: Bug
• Resolution: Incomplete
• Priority: Critical
• Fix Version/s: Lustre 2.11.0
• Affects Version/s: Lustre 2.10.0
• Labels: zfs
• Environment: CentOS 7.3 - 3.10.0-514.26.2.el7.x86_64; Lustre 2.10.0; ZFS 0.7.0

Epic/Theme

zfs

• Severity: 3
• Assignee: Alex Zhuravlev
• Reporter: Christopher Johnston (Inactive)

Description

At my company we use mmap() exclusively for accessing our data on Lustre. For starters we are seeing some very weird (or maybe expected) poor random read/write performance for these types of access patterns. I decided to give Lustre 2.10.0 a try with ZFS 0.7.0 as the backend instead of ldisk and after compiling and building the RPMs, the filesystem mounted up just fine. I then started doing some iozone runs to test the stability of the filesystem and although izone does complete uts benchmark I am seeing a lot of stack traces coming out of various kernel threads. Note I am only seeing this when using mmap(). We also ran our application as well just to verify. I am also going to try with an ldiskfs format as well to see if this changes anything.

My ZFS settings are modest, with 50% of memory allocated to the ARC:

options zfs zfs_arc_max=3921674240 zfs_prefetch_disable=1
recordsize=1M
compression=on
dedupe=off
xattr=sa
dnodesize=auto

Below is the output from the stack trace:

Aug  8 09:38:04 dev-gc01-oss001 kernel: BUG: Bad page state: 87 messages suppressed
Aug  8 09:38:04 dev-gc01-oss001 kernel: BUG: Bad page state in process socknal_sd00_01  pfn:1cbac1
Aug  8 09:38:04 dev-gc01-oss001 kernel: page:ffffea00072eb040 count:0 mapcount:-1 mapping:          (null) index:0x0
Aug  8 09:38:04 dev-gc01-oss001 kernel: page flags: 0x2fffff00008000(tail)
Aug  8 09:38:04 dev-gc01-oss001 kernel: page dumped because: nonzero mapcount
Aug  8 09:38:04 dev-gc01-oss001 kernel: Modules linked in: 8021q garp mrp stp llc osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_zfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg i2c_piix4 parport_pc i2c_core parport pcspkr nfsd nfs_acl lockd grace binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c zfs(POE) zunicode(POE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) zlib_deflate sd_mod crc_t10dif crct10dif_generic virtio_net virtio_scsi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw virtio_pci virtio_ring virtio
Aug  8 09:38:04 dev-gc01-oss001 kernel: CPU: 0 PID: 2558 Comm: socknal_sd00_01 Tainted: P    B      OE  ------------   3.10.0-514.26.2.el7.x86_64 #1
Aug  8 09:38:04 dev-gc01-oss001 kernel: Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Aug  8 09:38:04 dev-gc01-oss001 kernel: ffffea00072eb040 00000000005e265e ffff8800b879f5a8 ffffffff81687133
Aug  8 09:38:04 dev-gc01-oss001 kernel: ffff8800b879f5d0 ffffffff81682368 ffffea00072eb040 0000000000000000
Aug  8 09:38:04 dev-gc01-oss001 kernel: 000fffff00000000 ffff8800b879f618 ffffffff8118946d fff00000fe000000
Aug  8 09:38:04 dev-gc01-oss001 kernel: Call Trace:
Aug  8 09:38:04 dev-gc01-oss001 kernel: [<ffffffff81687133>] dump_stack+0x19/0x1b
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff81682368>] bad_page.part.75+0xdf/0xfc
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff8118946d>] free_pages_prepare+0x16d/0x190
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff811897b9>] __free_pages_ok+0x19/0xd0
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff8118988b>] free_compound_page+0x1b/0x20
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff81683526>] __put_compound_page+0x1f/0x22
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff81683698>] put_compound_page+0x16f/0x17d
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff8118edfc>] put_page+0x4c/0x60
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff8155ec1f>] skb_release_data+0x8f/0x140
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff8155ecf4>] skb_release_all+0x24/0x30
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff8155f1ec>] consume_skb+0x2c/0x80
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff8156f06d>] __dev_kfree_skb_any+0x3d/0x50
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffffa0019afb>] free_old_xmit_skbs.isra.32+0x6b/0xc0 [virtio_net]
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffffa0019baf>] start_xmit+0x5f/0x4f0 [virtio_net]
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff8156f9a1>] dev_hard_start_xmit+0x171/0x3b0
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff81597574>] sch_direct_xmit+0x104/0x200
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff8157252c>] __dev_queue_xmit+0x23c/0x570
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff81572870>] dev_queue_xmit+0x10/0x20
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff815b6876>] ip_finish_output+0x466/0x750
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff815b7873>] ip_output+0x73/0xe0
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff815b5531>] ip_local_out_sk+0x31/0x40
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff815b58a3>] ip_queue_xmit+0x143/0x3a0
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff815cf04f>] tcp_transmit_skb+0x4af/0x990
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff815cf68a>] tcp_write_xmit+0x15a/0xce0
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff815d048e>] __tcp_push_pending_frames+0x2e/0xc0
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff815bed2c>] tcp_push+0xec/0x120
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff815c25b8>] tcp_sendmsg+0xc8/0xc40
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff815ed854>] inet_sendmsg+0x64/0xb0
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff81555ff0>] sock_sendmsg+0xb0/0xf0
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff8168ea3b>] ? _raw_spin_unlock_bh+0x1b/0x40
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff81556067>] kernel_sendmsg+0x37/0x50
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffffa09f40d9>] ksocknal_lib_send_iov+0xd9/0x140 [ksocklnd]
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffffa09ed32f>] ksocknal_process_transmit+0x2af/0xb90 [ksocklnd]
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffffa09f1b84>] ksocknal_scheduler+0x204/0x670 [ksocklnd]
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff810b1b20>] ? wake_up_atomic_t+0x30/0x30
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffffa09f1980>] ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd]
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff810b0a4f>] kthread+0xcf/0xe0
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff81697758>] ret_from_fork+0x58/0x90
Aug  8 09:38:05 dev-gc01-oss001 kernel: [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140

-Chris

Issue Links

• is related to: LU-9305 Running File System Aging create write checksum errors

Comments

[Christopher Johnston (Inactive)]

Sorry for the delay in response, other priorities at work took precedence. This is still likely an issue and I have not tried to use 2.10.1 which is recently released but there was still an issue with my test cluster when we tried it recently. I will look to stand up a latest and greatest version over the weekend and retest as using ZFS is becoming more important for me to take advantage of the compression and snapshots.

[Peter Jones]

ok this does not seem to be a priority for you so we'll just close the ticket. If someone else encounters this and wants to engage we can reopen the ticket

[Peter Jones]

chjohnst are you able to supply some more detail here?

[Alex Zhuravlev]

Hi, did you get these traces on the client side?

[Peter Jones]

Alex

Is this related to the LU-9305 change?

Peter