[#LU-11848] filefrag/FIEMAP doesn't work for PFL or FLR files

It appears that FIEMAP doesn't work quite correctly for PFL or FLR files. I'd expect that it would dump the FIEMAP information for all copies of a mirrored file, but it exits when the first object is printed. I verified with GDB that the fiemap structure returned by the kernel only has a single extent in it.

$ lfs getstripe /mnt/testfs/flr2
/mnt/testfs/flr2
  lcm_layout_gen:    5
  lcm_mirror_count:  2
  lcm_entry_count:   2
    lcme_id:             65537
    lcme_mirror_id:      1
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 3
      lmm_objects:
      - 0: { l_ost_idx: 3, l_fid: [0x100030000:0x4:0x0] }

    lcme_id:             131074
    lcme_mirror_id:      2
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 0
      lmm_objects:
      - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x4:0x0] }

$ filefrag -v /mnt/testfs/flr2
Filesystem type is: bd00bd0
File size of /mnt/testfs/flr2 is 8388608 (8192 blocks of 1024 bytes)
 ext:     device_logical:        physical_offset: length:  dev: flags:
   0:        0..    8191:     140000..    148191:   8192: 0003: last,net,eof
/mnt/testfs/flr2: 1 extent found

For FLR files this is at least somewhat usable, since a full copy of the file is handled. However, for PFL files, the kernel returns an -EFBIG error after the first component is read:

$ lfs getstripe /mnt/testfs/pfl3x8
  lcm_entry_count:   3
    lcme_extent.e_start: 0
    lcme_extent.e_end:   1048576
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_objects:
      - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x2:0x0] }

    lcme_extent.e_start: 1048576
    lcme_extent.e_end:   4194304
      lmm_stripe_count:  3
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_objects:
      - 0: { l_ost_idx: 3, l_fid: [0x100030000:0x2:0x0] }
      - 1: { l_ost_idx: 0, l_fid: [0x100000000:0x2:0x0] }
      - 2: { l_ost_idx: 1, l_fid: [0x100010000:0x2:0x0] }

    lcme_extent.e_start: 4194304
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  4
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_objects:
      - 0: { l_ost_idx: 3, l_fid: [0x100030000:0x3:0x0] }
      - 1: { l_ost_idx: 0, l_fid: [0x100000000:0x3:0x0] }
      - 2: { l_ost_idx: 1, l_fid: [0x100010000:0x3:0x0] }
      - 3: { l_ost_idx: 2, l_fid: [0x100020000:0x3:0x0] }

$ filefrag -v /mnt/testfs/pfl3x8
Filesystem type is: bd00bd0
File size of /mnt/testfs/pfl3x8 is 8388608 (8192 blocks of 1024 bytes)
 ext:     device_logical:        physical_offset: length:  dev: flags:
   0:        0..    1023:     134880..    135903:   1024: 0002: net
   1:        0..    1023:     134880..    135903:   1024: 0000: net
   2:     1024..    2047:     136928..    137951:   1024: 0003: net

It starts almost correctly, with an extent on OST0002 for [0,1MB] and then something that looks like a duplicate of the first extent with only the device number changed, and an extent on OST0003 for [1MB,2MB]. The last 6 extents are not returned in the first call despite there being plenty of room in the fiemap array.

Running strace shows:

ioctl(3, FS_IOC_FIEMAP, {fm_start=0, fm_length=18446744073709551615, fm_flags=0x40000000 /* FIEMAP_FLAG_??? */, fm_extent_count=292} => {fm_flags=0x40000000 /* FIEMAP_FLAG_??? */, fm_mapped_extents=3, fm_extents=...}) = 0
ioctl(3, FS_IOC_FIEMAP, {fm_start=0, fm_length=18446744073709551615, fm_flags=0x40000000 /* FIEMAP_FLAG_??? */, fm_extent_count=292}) = -1 EFBIG (File too large)

Running under GDB and dumping the returned extents shows that indeed 3 separate extents are returned (so strangeness is not in filefrag), and then -EFBIG is returned on the second ioctl(FIEMAP) call when it tries to continue on the next extent.

247                     rc = ioctl(fd, FS_IOC_FIEMAP, (unsigned long) fiemap);
248                     if (rc < 0) {
(gdb) p *fiemap
$1 = {fm_start = 0, fm_length = 18446744073709551615, fm_flags = 1073741824, 
  fm_mapped_extents = 3, fm_extent_count = 292, fm_reserved = 0, 
  fm_extents = 0x7fffffff9ca0}
(gdb) p fiemap->fm_extents[0]
$2 = {fe_logical = 0, fe_physical = 138117120, fe_length = 1048576, 
  fe_reserved64 = {0, 0}, fe_flags = 2147483648, fe_device = 2, fe_reserved = {0, 0}}
(gdb) p fiemap->fm_extents[1]
$3 = {fe_logical = 0, fe_physical = 138117120, fe_length = 1048576, 
  fe_reserved64 = {0, 0}, fe_flags = 2147483648, fe_device = 0, fe_reserved = {0, 0}}
(gdb) p fiemap->fm_extents[2]
$4 = {fe_logical = 1048576, fe_physical = 140214272, fe_length = 1048576, 
  fe_reserved64 = {0, 0}, fe_flags = 2147483648, fe_device = 3, fe_reserved = {0, 0}}
:
:
305                     if (flags & FIEMAP_FLAG_DEVICE_ORDER) {
306                             fm_ext[0].fe_logical =  fm_ext[i - 1].fe_logical + fm_ext[i - 1].fe_length;
308                             fm_ext[0].fe_device =   fm_ext[i - 1].fe_device;
309                             fiemap->fm_start =      0;
:
:
244                     fiemap->fm_length = ~0ULL;
245                     fiemap->fm_flags = flags;
246                     fiemap->fm_extent_count = count;
247                     rc = ioctl(fd, FS_IOC_FIEMAP, (unsigned long) fiemap);
248                     if (rc < 0) {
251                             rc = -errno;
(gdb) p *fiemap
$7 = {fm_start = 0, fm_length = 18446744073709551615, fm_flags = 1073741824, 
  fm_mapped_extents = 0, fm_extent_count = 292, fm_reserved = 0, 
  fm_extents = 0x7fffffff9ca0}
(gdb) p rc
$8 = -27

It looks like the LOV code needs to be taught how to iterate over a composite layout to populate the FIEMAP structure, and to be able to find the right component to restart. It may be that we need to make changes to filefrag itself, since it currently only returns the file logical offset of the end of the last printed extent and the device number it was on. This could be ambiguous in rare cases if there were two mirrors on the same OST for the same file offset and FIEMAP was interrupted just there, but it doesn't seem likely and is not a primary concern.

[LU-11848] filefrag/FIEMAP doesn't work for PFL or FLR files Created: 10/Jan/19 Updated: 12/Sep/23 Resolved: 18/Dec/20
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 2.11.0, Lustre 2.12.0
Fix Version/s:	Lustre 2.14.0

[LU-11848] filefrag/FIEMAP doesn't work for PFL or FLR files Created: 10/Jan/19 Updated: 12/Sep/23 Resolved: 18/Dec/20