[LU-9982] Clients striping from mapped FID in nodemap

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Labels: cea
• Assignee: Etienne Aujames
• Reporter: Jean-Baptiste Riaux (Inactive)

Description

The main idea is to bind a FID given by path or FID (formatted with or without brackets)
The associated set of clients (range) when creating files or directories, will inherit striping information from template FID when it's possible. Striping priorities:

1. Explicit striping
2. Parent striping
3. Template striping
4. Filesystem default

As it's possible to query the nodemap on a given export, it's possible to get the information on the MDT stack which will transmit the template object to lower layer. Then, when a template exists, a new field into dt_allocation_hint points to the template object and will be read in lod_ah_init function and lod_layout_component is filled accordingly.

Example of usage:

• Start lustre
• On a client (normal striping)

  [root@client ~]# mount -t lustre 192.168.10.134@tcp:/testfs /lustre/testfs2
  [root@client ~]# cd /lustre/testfs2/
  [root@client testfs2]# echo "test" > testfile
  [root@client testfs2]# lfs getstripe -c testfile
  1
  [root@client testfs2]# lfs getstripe -c my_template
  2
  [root@client testfs2]# lfs path2fid my_template
  [0x200000402:0x1:0x0]
  

• On the MGS node (creating the nodemap + adding client or range)

  [root@mds1]# lctl nodemap_add  SG1
  [root@mds1]# lctl nodemap_add_range --name SG1 --range 192.168.10.[130-140]@tcp
  [root@mds1]# lctl set_param -P nodemap.SG1.template=0x200000401:0x1:0x0
  or
  [root@mds1]# lctl nodemap_set_template --name SG1 --template /lustre/testfs2/my_template
  or
  [root@mds1]# lctl nodemap_set_template --name SG1 --template 0x200000401:0x1:0x0
  or
  [root@mds1]# lctl nodemap_set_template --name SG1 --template "[0x200000401:0x1:0x0]"
  [root@mds1]# lctl nodemap_modify --name SG1 --property admin --value 1 # working as root
  [root@mds1]# lctl nodemap_activate 1
  [root@mds1]# cat /proc/fs/lustre/nodemap/SG1/template
  0x200000402:0x1:0x0
  

• On the client:

  [root@client testfs2]# echo "test" > testfile2
  [root@client testfs2]# lfs getstripe -c testfile2
  2
  

Todo in next days:

• add a valid fid format check when bound
• I intend to change the type of record from char[FID_LEN] to lu_fid directly (add a nodemap record). It will prevent the call to sscanf in mdt_reint_open

Issue Links

• is related to: LU-11656 "lfs getstripe" on directory does not show default root layout
• is related to: LU-11510 preserve PFL/FLR/DoM layout with lfs_migrate
• is related to: LU-11586 sanity-pfl test_10: FAIL: /mnt/lustre/d10.sanity-pfl/file2 does not inherite root layout (subdiretory mounts)
• is related to: LU-10345 allow specifying file layout by FID and filename extension
• is related to: LU-11234 Choose lustre pool to place file’s objects by user-defined policies
• is related to: LU-11739 Don't inherit default layout from root for new subdirectories
• is related to: LU-17775 Allow 'lfs setstripe --copy FID' to inherit layout from FID

Comments

[Andreas Dilger]

What would be a useful enhancement to this is defaulting to use the root directory of the fileset as the template. That would work the same way as the current root directory of the whole filesystem, making the virtualization of Lustre more transparent.

[Jean-Baptiste Riaux (Inactive)]

Performance impact measures (worst case is 1.9% degradation with feature enabled and used, multiple runs done).

I will add the performance impact in my next commit message.

Creating 500K files in a directory on a single node setup.

	Master	LU-9982	%diff
stripe count 1	33.245s	33.770s	+1.579%
stripe count 2	33.865s	34.520s	+1.934%

Also found a bug when bursting files creation with SELinux enforcing (no problem when SELinux is set to permissive). Currently investigating (lod_ah_init).

+ 1 minor bug to fix (memory leak when changing template, nodemap->fidtpl not freed). 

[Jean-Baptiste Riaux (Inactive)]

Document design updated. 

Code updated (memory leaks fixed due to LU-8998 landing, PFL) + other fixes (leak when nodemap was destroyed and nodemap_putref calls added when missing) + added stripe offset and ost-list support for the template (see lod_ah_init in patch 28972).

I am really interested by feedback on lod_ah_init in file lod_object.c because I am pretty sure there is a smarter or more delicate way to transfer template lod_layout_component into created object.

[Jean-Baptiste Riaux (Inactive)]

Added tests in file sanity-nodemap.sh (dependency on https://review.whamcloud.com/#/c/28972)

[Gerrit Updater]

Jean-Baptiste Riaux ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31171
Subject: LU-9982 lustre: Clients striping from mapped FID in nodemap
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: fba1cf5a2e8bf664d2caa430073f34b2805703c3

[Jean-Baptiste Riaux (Inactive)]

New version of the patch available (still https://review.whamcloud.com/#/c/28972) using dt_allocation_hint.