[LU-5746] llapi: fid2path doesn't return all the names associated with a fid

• Status: Resolved
• Project: Lustre
• Type: Bug
• Priority: Minor
• Reporter: Frank Zago (Inactive)
• Assignee: Andreas Dilger
• Resolution: Not a Bug
• Severity: 3

Description

llapi_fid2path can (or should) enumerate all the files associated with a fid, by using the linkno parameter. This is supported by lfs.

However when the number of hard link is big, this is no longer true. For instance, create a file, and many hardlinks to it:

cd /mnt/lustre/test/
touch foo
count=1; while [[ $count -le 180 ]]; do ln foo alinktofoo$count; (( count++ )) ; done

Get the fid of the file

# ~/lustre-cleanup/lustre/utils/lfs path2fid ./foo
[0x200000400:0x8b8:0x0]

and then call fid2path on it:

# ~/lustre-cleanup/lustre/utils/lfs fid2path /mnt/lustre/ 0x200000400:0x8b8:0x0

The output will be something like

/mnt/lustre/test/foo
/mnt/lustre/test/alinktofoo1
/mnt/lustre/test/alinktofoo2
/mnt/lustre/test/alinktofoo3
/mnt/lustre/test/alinktofoo4
.......
/mnt/lustre/test/alinktofoo130
/mnt/lustre/test/alinktofoo131
/mnt/lustre/test/alinktofoo132

Only the first 133 entries are returned, leaving out 47.

The interesting thing is that if the names of the links are longer (replace alinktofoo$count with averylinktofoo$count), then even less names will be returned.

Potential impact on existing programs is unknown.

Comments

[Patrick Farrell (Inactive)]

I'm mostly just free associating, but is there any link to the limit/optimization described by Andreas at the end of the comments here:
https://jira.hpdd.intel.com/browse/LU-2490

[Frank Zago (Inactive)]

In a similar testing, if the length of the link names are 8 characters, 158 entries are returned. As the length grows, less entries are returned. With 63 characters, it's 55 entries. With 83, it's 43 entries. At the maximum length (255), only 13 entries are returned.

The length of the leading path doesn't appear to have any influence.

[Andreas Dilger]

As described in LU-2490 this is essentially working as designed. The last component of the filename as well as the parent directory FID are stored in each link xattr, so longer filenames will result in fewer entries:

The limit is only for the number of reverse name entries for a single file, stored in the "link" xattr. This is useful for lfs fid2path to generate pathnames from a FID for error reporting or lustre_rsync.

Beyond the 128 hard link count, the reverse links are dropped, though not as silently as they should be. The limit is due to efficiency - since an xattr is completely searched and rewritten for each link added and removed, it gets into an O(n^2) behaviour if there are too many reverse links. Keeping 128 links was decided as an acceptable real-world usage, and for cases like simul or similar it will still work with lots of links. I think we're increasing the hard link limit in the "link" xattr as part of the LFSCK changes, because we now allow large xattrs, but I don't see it ever being unlimited due to O(n^2) slowdowns for updating the xattr.

If the large_xattr feature is enabled on the MDS, it is able to store more links, but this comes with a noticeable performance cost as the link count increases. In our surveys of HPC filesystems[*] there are rarely more than a handful of links on regular files, and having more than a hundred links to a file is basically nonexistent.

Is this a real problem on a real system, or just something that you found while testing corner conditions?

[*] http://www.pdsi-scidac.org/fsstats/ has some results, and we've asked users to run this tool on their filesystems and send us the output along with "lfs df" and "lfs df -i".

[Frank Zago (Inactive)]

This is something I found while writing some test code for the fid API.
So if it works as designed, you can close this ticket.