[LU-13151] Add parent FID to MTIME changelogs Created: 17/Jan/20 Updated: 13/Jan/23 Resolved: 08/Sep/20 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.13.0 |
| Fix Version/s: | Lustre 2.14.0 |
| Type: | Improvement | Priority: | Minor |
| Reporter: | Nikitas Angelinas | Assignee: | Nikitas Angelinas |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Issue Links: |
|
||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||
| Description |
|
Some tools that maintain copies of filesystem metadata in an external database monitor changelogs for changes to the filesystem, in order to determine files that need to be rescanned. This can result in a large number of small updates to the external database that can reduce the tool's ingest performance. It might be beneficial to instead track and scan complete directories that contain modified files and update the external database using bulk operations. Adding the parent FID to MTIME changelogs allows to more efficiently determine the parent directories for some types of file data modifications, by issuing OBD_IOC_FID2PATH once for each parent FID, instead of once for each file FID. |
| Comments |
| Comment by Gerrit Updater [ 17/Jan/20 ] |
|
Nikitas Angelinas (nangelinas@cray.com) uploaded a new patch: https://review.whamcloud.com/37264 |
| Comment by Nikitas Angelinas [ 17/Jan/20 ] |
|
The patch submitted uses the link EA to add the parent FID in the MDT; we have not verified whether this affects performance negatively. We think this patch could help in detecting some types of file data modifications, but not all, as MTIME changelogs don't seem to be generated e.g. when appending to a file. |
| Comment by Nathan Rutman [ 19/Jan/20 ] |
|
In particular, code like GUFI can take a "suspects" list of modified directories to rescan. Without the parent fid in the changelog, each mtime record needs a fid2path to find the parent directory to rescan. With the parent in the changelog, all the information except the parent can be thrown away, and all records "collapse" into a small set of modified directories. |
| Comment by Gerrit Updater [ 08/Sep/20 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/37264/ |
| Comment by Peter Jones [ 08/Sep/20 ] |
|
Landed for 2.14 |