[LU-12499] improve the file path layout of HSM posix copytool

Wed, 3 Jul 2019 15:40:15 +0000

Andreas's comment on the current layout:

The problem with having the low bits in the top of the path is that this will create a very deep directory tree, and there will only be a single entry in each of the leaf directories because the f_seq values never repeat. Also, the f_seq values will be constant for a client for 128K creates until a client gets a new one, so that isn't giving you any fan-out, so those levels of the tree should just be dropped.

This is a serious design mistake in the POSIX copytool that we shouldn't repeat here.

Instead, it probably makes sense to have a 2^16 fanout by f_oid at the top level, and then store the FID in the leaf. That will give 16 files/leaf per million files created by this client (== 16K files/leaf per billion creates), and avoids all contention (there are not going to be 64K threads active on this client). Currently, "f_oid >> 16" will mostly be 0 or sometimes 1, so it doesn't provide much fan-out.}}

A good layout would be:

(fid->f_oid & oxFFFF)/FID

Whamcloud Community JIRA

[LU-12499] improve the file path layout of HSM posix copytool