[LU-10329] DNE3: REMOTE_PARENT_DIR scalability Created: 05/Dec/17  Updated: 10/Aug/23

Status: Open
Project: Lustre
Component/s: None
Affects Version/s: None
Fix Version/s: Lustre 2.16.0

Type: Improvement Priority: Critical
Reporter: Andreas Dilger Assignee: Lai Siyao
Resolution: Unresolved Votes: 0
Labels: LMR, dne3, medium

Issue Links:
Gantt End to Start
has to be done before LU-11025 DNE3: directory restripe Resolved
has to be done before LU-10784 DNE3: mkdir() automatically create re... Resolved
Related
is related to LU-12892 Large directory feature is not enable... Resolved
is related to LU-11546 enable large_dir support for MDTs Resolved
is related to LU-10192 Agent entry for cross-MDTs reference Resolved
is related to LU-4684 DNE3: allow migrating DNE striped dir... Resolved
is related to LU-15383 DNE directories not connected to REMO... Open
is related to LU-4876 LFSCK remove entry from /REMOTE_PAREN... Open
is related to LU-11446 ldiskfs inodes nlink mismatch with DNE Open
is related to LU-11502 add striped directory migration suppo... Open
is related to LU-15330 ext2fs_get_pathname() very slow for l... Open
is related to LU-12310 MDT Device-level Replication/Mirroring Open
is related to LU-15307 Option to set max directory stripe co... Resolved
is related to LU-13440 DNE3: limit directory default layout ... Resolved
Rank (Obsolete): 9223372036854775807
Epic Link: MDT rebalance v3

 Description   

For DNE filesystems where there are large numbers of remote entries, for example once LU-4684 is landed to restripe directories, or DNE2 striped directories with many renames, the size of the REMOTE_PARENT_DIR may become very large.

In order to limit contention and scaling issues in REMOTE_PARENT_DIR it makes sense to have multiple such directories. As a starting point, one REMOTE_PARENT_DIR_MDTxxxx for each remote MDT would be useful, but it may be necessary to have a tree of directories similar to the O/<seq>/dN object directories. Having a separate REMOTE_PARENT_DIR_MDTxxxx per MDT would also allow LFSCK to efficiently scan remote entries for a given MDT, if there was a problem (e.g. MDT was marked offline and returned into the namespace later).

 Comments   
Comment by Andreas Dilger [ 18/Oct/18 ]

This is needed before LU-11025 or LU-10784 or implements automatic directory restriping or remote directories. Otherwise, the large number of remote entries in the filesystem will exceed the per-directory limit (unless LU-11546 is implemented first, and that is a sub-optimal solution).

Comment by Andreas Dilger [ 30/Jul/20 ]

It seems that the large directory patches have landed, so the need for this is reduced.

However, it would still be useful to have a per-MDT REMOTE_PARENT_DIR_MDTxxxx directory, since that would simplify things like LFSCK checking if one MDT was offline/corrupted/removed. It also provides a point of scaling so that there can be different updates between pairs of MDTs that do not contend for the same locks.

Comment by Andreas Dilger [ 02/Dec/21 ]

In addition to improving performance by having multiple REMOTE_PARENT_DIR_MDTxxxx directories, another significant benefit would be reduced risk from filesystem corruption. An MDT with 60M entries in REMOTE_PARENT_DIR is currently running e2fsck because of a problem with that directory, with an unknown ETA, so having multiple independent directories would reduce risk and the number of entries that need to be repaired significantly, and potentially improve the scalability of parallel pass2/pass3 operations as well.

Comment by Andreas Dilger [ 03/Apr/23 ]

For upgrading MDTs that are currently using a single REMOTE_PARENT_DIR, I think a two-stage approach could be used. Initially, add "read-only" support for such directories into 2.16.0 to allow for downgrade:

  • accept OBD_INCOMPAT_REMOTE_PARENTS = 0x00000800 in tgt_scd, but do not set it
  • open REMOTE_PARENT_DIR_MDTxxxx directories when a remote MDT connection is made, using local_file_find(), if available
  • do agent FID lookups in REMOTE_PARENT_DIR first, and fall back to REMOTE_PARENT_DIR_MDTxxxx if it exists and FID was not found
  • create new entries in REMOTE_PARENT_DIR only (as before)

Then in 2.17+ start using those directories (which would be incompatible for pre-2.16 servers, or servers not patched as above):

  • set OBD_INCOMPAT_REMOTE_PARENTS = 0x00000800 in tgt_scd so that unpatched MDS cannot mount the MDT
  • create/open REMOTE_PARENT_DIR_MDTxxxx directories when a remote MDT connection is made, using local_file_find_or_create()
  • do agent FID lookups in REMOTE_PARENT_DIR_MDTxxxx first, and fall back to REMOTE_PARENT_DIR if it exists and FID was not found
  • start linking new agent inodes into REMOTE_PARENT_DIR_MDTxxxx based on remote parent FID
  • update LFSCK to do the same when linking or looking up remote FIDs on the MDT
  • (optionally?) start a background thread to move entries from REMOTE_PARENT_DIR to the proper ..._MDTxxxx subdir based on the parent FID, then remove the old REMOTE_PARENT_DIR when it is empty.

Having a two-stage update process like this, and minimizing the "read-only" patches to simplify backport (e.g. to b2_15) will allow upgrade/downgrade without breaking the whole filesystem.

One minor drawback of having separate directories would be that renaming files/subdirs from one remote MDT directory to another would also mean renaming the agent locally from one REMOTE_PARENT_DIR_MDTxxxx to another, but that is not worse than a local cross-directory rename, and only a small fraction of the overhead of the distributed rename itself (BFL, multiple MDT transactions, etc.). That local filesystem overhead is probably offset by not having a single huge REMOTE_PARENT_DIR to hold all of the remote links.

Comment by Lai Siyao [ 08/May/23 ]

REMOTE_PARENT_DIR is used in osd layer only, which means it doesn't know which MDT its parent is located, but only knows that whether parent is on local MDT. So we can't create a subdir for each MDT, do you think it's okay to create 64 subdirs for all remote MDTs?

Comment by Andreas Dilger [ 08/May/23 ]

There are definitely benefits to splitting the entries up by remote MDT. This could be used to quickly find/check/remove remote links for a specific MDT if needed (eg. MDT removed/lost with metadata redundancy).

Does it make sense to lift REMOTE_PARENT_DIR out of the OSD and into MDD?

Comment by Gerrit Updater [ 10/Aug/23 ]

"Lai Siyao <lai.siyao@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51910
Subject: LU-10329 obdclass: tidy up linkEA code
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b27fd4006b7e763da72eb9d834fd4423293e5349

Comment by Gerrit Updater [ 10/Aug/23 ]

"Lai Siyao <lai.siyao@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51911
Subject: LU-10329 obdclass: add linkEA version 2
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8ea62123db1a9bbbb2e36b511c03fbd368f80b75

Comment by Gerrit Updater [ 10/Aug/23 ]

"Lai Siyao <lai.siyao@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51912
Subject: LU-10329 osd-ldiskfs: add osd_obj_map for remote objects
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ffbc774053bec88e1aacb16831b6f0fa632529d5

Generated at Sat Feb 10 02:34:04 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.