[LU-18866] ZFS OSD is seeing timeouts with fzap_lookup

• Type: Bug
• Resolution: Unresolved
• Priority: Major
• Affects Version/s: Lustre 2.15.5
• Labels: ORNL
• Environment: RHEL 8 running 4.18.0-513.24 and ZFS version 2.1.15
• Severity: 3
• Assignee: WC Triage
• Reporter: James A Simmons

Description

Seeing the following:

*kernel: Lustre: ll_ost05_032: service thread pid 323956 was inactive for 202.979 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:* 
*kernel: Lustre: ll_ost05_032: service thread pid 323956 was inactive for 202.979 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:* 
*kernel: Pid: 323956, comm: ll_ost05_032 4.18.0-513.24.1.el8_9.x86_64 #1 SMP Thu Mar 14 14:20:09 EDT 2024* 
*kernel: Pid: 323956, comm: ll_ost05_032 4.18.0-513.24.1.el8_9.x86_64 #1 SMP Thu Mar 14 14:20:09 EDT 2024* 
*kernel: Call Trace TBD:* 
*kernel: Call Trace TBD:* 
*kernel: [<0>] __cv_timedwait_common+0x132/0x160 [spl]* 
*kernel: [<0>] __cv_timedwait_common+0x132/0x160 [spl]* 
*kernel: [<0>] __cv_timedwait_io+0x15/0x20 [spl]* 
*kernel: [<0>] __cv_timedwait_io+0x15/0x20 [spl]* 
*kernel: [<0>] zio_wait+0x133/0x270 [zfs]* 
*kernel: [<0>] zio_wait+0x133/0x270 [zfs]* 
*kernel: [<0>] dbuf_read+0x28c/0x580 [zfs]* 
*kernel: [<0>] dbuf_read+0x28c/0x580 [zfs]* 
*kernel: [<0>] dbuf_hold_impl+0x484/0x630 [zfs]* 
*kernel: [<0>] dbuf_hold_impl+0x484/0x630 [zfs]* 
*kernel: [<0>] dbuf_hold+0x2c/0x60 [zfs]* 
*kernel: [<0>] dbuf_hold+0x2c/0x60 [zfs]* 
*kernel: [<0>] dmu_buf_hold_by_dnode+0x5a/0x110 [zfs]* 
*kernel: [<0>] dmu_buf_hold_by_dnode+0x5a/0x110 [zfs]* 
*kernel: [<0>] zap_get_leaf_byblk.isra.13+0x62/0x250 [zfs]* 
*kernel: [<0>] zap_get_leaf_byblk.isra.13+0x62/0x250 [zfs]* 
*kernel: [<0>] zap_deref_leaf+0xa1/0xf0 [zfs]* 
*kernel: [<0>] zap_deref_leaf+0xa1/0xf0 [zfs]* 
*kernel: [<0>] fzap_add_cd+0x5e/0x180 [zfs]* 
*kernel: [<0>] fzap_add_cd+0x5e/0x180 [zfs]* 
*kernel: [<0>] fzap_add+0x5a/0x80 [zfs]* 
*kernel: [<0>] fzap_add+0x5a/0x80 [zfs]* 
*kernel: [<0>] zap_add_impl+0xf9/0x250 [zfs]* 
*kernel: [<0>] zap_add_impl+0xf9/0x250 [zfs]* 
*kernel: [<0>] zap_add+0x78/0xb0 [zfs]* 
*kernel: [<0>] zap_add+0x78/0xb0 [zfs]* 
*kernel: [<0>] osd_create+0xa17/0xcf0 [osd_zfs]* 
*kernel: [<0>] osd_create+0xa17/0xcf0 [osd_zfs]* 
*kernel: [<0>] ofd_precreate_objects+0x141d/0x2000 [ofd]* 
*kernel: [<0>] ofd_precreate_objects+0x141d/0x2000 [ofd]* 
*kernel: [<0>] ofd_create_hdl+0x820/0x1870 [ofd]* 
*kernel: [<0>] ofd_create_hdl+0x820/0x1870 [ofd]* 
*kernel: [<0>] tgt_request_handle+0xccd/0x1a20 [ptlrpc]* 
*kernel: [<0>] tgt_request_handle+0xccd/0x1a20 [ptlrpc]* 
*kernel: [<0>] ptlrpc_server_handle_request+0x323/0xbe0 [ptlrpc]* 
*kernel: [<0>] ptlrpc_server_handle_request+0x323/0xbe0 [ptlrpc]* 
*kernel: [<0>] ptlrpc_main+0xbec/0x1530 [ptlrpc]* 
*kernel: [<0>] ptlrpc_main+0xbec/0x1530 [ptlrpc]*
**