[LU-8927] osp-syn processes contending for osq_lock drives system cpu usage > 80%

• Type: Bug
• Resolution: Won't Fix
• Priority: Major
• Labels: llnl zfs
• Environment: lustre-2.8.0_5.chaos-2.ch6.x86_64; zfs-0.7.0-0.6llnl.ch6.x86_64; DNE with 16 MDTs
• Severity: 3
• Assignee: Alex Zhuravlev
• Reporter: Olaf Faaland

Description

Ran jobs which created remote directories (not striped) and then ran mdtest within them, several MDS nodes are using >80% of their cpu time for osp-syn-* processes.

There are 36 osp-syn-* processes.

The processes are spending almost all their time contending for osq_lock. According to perf, the offending stack is:

osq_lock
__mutex_lock_slowpath
mutex_lock
spa_config_enter
bp_get_dsize
dmu_tx_hold_free
osd_declare_object_destroy
llog_osd_declare_destroy
llog_declare_destroy
llog_cancel_rec
llog_cat_cancel_records
osp_sync_process_committed
osp_sync_process_queues
llog_process_thread
llog_process_or_fork
llog_cat_process_cb
llog_process_thread
llog_process_or_fork
llog_cat_process_or_fork
llog_cat_process
osp_sync_thread
kthread
ret_from_fork
osp-syn-X-Y

Issue Links

• is related to: LU-2435 inode accounting in osd-zfs is racy
• is related to: LU-8873 use sa_handle_get_from_db()
• is related to: LU-8882 osd-zfs to use bynode methods
• is related to: LU-8928 osd-zfs should use dnode_t instead of dbuf