[LU-17307] osd_dirent_count() keeps multiple threads busy

Related — Fri, 15 Dec 2023 19:17:27 +0000

When osd_attr_get() is accessing a directory for the first time, it calls osd_dirent_count() to iterate over the directory and fill in obj->oo_dirent_count:

 iterate_dir+0x70/0x140
 osd_ldiskfs_it_fill+0xbd/0x290 [osd_ldiskfs]
 osd_it_ea_next+0xc2/0x150 [osd_ldiskfs]
 osd_attr_get+0x4bc/0x730 [osd_ldiskfs]
 lod_attr_get+0x8b/0x170 [lod]
 mdd_la_get+0x70/0x200 [mdd]
 mdd_attr_get+0x38/0x100 [mdd]
 mdt_attr_get_complex+0x4dd/0x800 [mdt]
 mdt_getattr_internal+0x445/0x1590 [mdt]
 mdt_getattr_name_lock+0x74d/0x2640 [mdt]
 mdt_intent_getattr+0x2a5/0x470 [mdt]
 mdt_intent_opc+0x1ba/0xb40 [mdt]
 mdt_intent_policy+0x1a9/0x370 [mdt]
 ldlm_lock_enqueue+0x3d4/0xb00 [ptlrpc]
 ldlm_handle_enqueue0+0x8b6/0x16d0 [ptlrpc]
 tgt_enqueue+0x62/0x220 [ptlrpc]
 tgt_request_handle+0x8bf/0x18c0 [ptlrpc]
 ptlrpc_server_handle_request+0x253/0xc40 [ptlrpc]
 ptlrpc_main+0xc88/0x26c0 [ptlrpc]
 kthread+0xd1/0xe0

However, there are several issues with this:

if the directory is very large (e.g. millions of entries), then this iteration can take a considerable time and blocks the MDS service thread until it is finished.
if multiple MDS threads are accessing the same directory, then all of the threads will try to count the number of entries in the directory, blocking all of the threads.
the oo_dirent_count value is only needed for auto directory split, which is not enabled today.

The entry counting should only be done if directory auto-split is enabled, avoiding overhead under normal cases. Since mdt_enable_dir_auto_split is controlled at the MDT level, the need for the count should be passed down to the OSD via an LA_DIRENT_COUNT valid flag. For osd-ldiskfs if this flag is not set then the count would be skipped. For osd-zfs it can optionally return the count since it is available for free.

If directory auto-split is enabled, it would be much more efficient to only have one thread do the directory iteration. This should be controlled by a flag in the object, and the other threads can ignore oo_dirent_count (return "0" or "LU_DIRENT_COUNT_UNSET" or the current number of entries found). At worst this would defer the auto split by a few entries, but that doesn't matter in the end since the thread doing the counting will always perform the check itself.

Whamcloud Community JIRA

[LU-17307] osd_dirent_count() keeps multiple threads busy