With the current mkdir QoS policy, once the directory level is deeper
than RR_MAX (max_inherit_rr), new subdirectories are created on the
parent MDT unless the MDTs' space/inode usage is very unbalanced. So a
busy MDT keeps getting new directories, and all the files later
created under them.
 
This ticket proposes to also consider MDT load during mkdir:
 
1. Round-robin (within RR_MAX) and space-imbalance placement are kept
as they are. If they choose the MDT, load is ignored.
 
2. Otherwise the new directory would stay on its parent MDT. The
client first checks whether the parent MDT is busy, using the
service time estimate the MDT already sends in every reply. If it
is clearly slower than the other MDTs (lmv.*.qos_load_threshold,
default 3 seconds), the directory goes to another MDT, preferring
less busy ones.
 
3. The MDS can also refuse a mkdir when it is overloaded (all service
threads busy and requests waiting longer than
mdt.*.mkdir_redirect_qwait_ms, disabled by default). The client
then creates the directory on an MDT on another server and avoids
the busy server for a few seconds. A mkdir is moved at most once,
and is never moved if there is no other server to go to.
 
MDTs on the same server share its load, so directories are only moved
between servers.