Details
-
Improvement
-
Resolution: Unresolved
-
Minor
-
None
-
Lustre 2.4.3
-
13934
Description
When our ZFS OSTs are heavily loaded we still can see the following console warning. Long ago we improved things such that this doesn't cause failures but it can limit create performance when there are a small number of OSTs in the filesystem.
2014-05-09 14:05:48 Lustre: lcz-OST0000: Slow creates, 4736/5120 objects created at a rate of 94/s 2014-05-09 14:05:48 Lustre: Skipped 5 previous similar messages
Recently I had a idea on how we might be able to fairly easily improve performance for precreate RPCs. One of the major reasons the creates can take a long time is because ofd_precreate_objects() calls ofd_object_find() serially to create the objects. The ofd_object_find() will result in at least one ZAP lookup and there's a decent chance that means performing a synchronous IO to disk. That alone will be slow but if the disk is already 100% utilized it can be very slow.
What occurred to me is that for the precreate case prefetching the objects we're about to do ofd_object_find()'s on should be very effective. A prefetch pass which called ofd_object_prefetch() prior to ofd_object_find() would allow us to effectively read all the needed ZAP blocks as quickly as possible. The subsequent ofd_object_find() calls would then have a good chance of being cache hits.
I'm proposing adding the following zap_prefetch() interface to ZFS to do what we need. Presumably this is something the ldiskfs OSD could optionally take advantage of as well. Obviously we'd still need to add a reasonable interface on the Lustre side.
https://github.com/zfsonlinux/zfs/pull/2318
Anyway, I wanted to file this to get your guys feedback and so I don't forget. It should be a pretty a straight forward improvement. It's also something I could see us making use of elsewhere in the code. As soon as we're aware of a FID we many need to lookup we could issue the prefetch which is entirely asynchronous.
Attachments
Issue Links
- is blocking
-
LU-2600 lustre metadata performance is very slow on zfs
-
- Resolved
-