Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Lustre 2.4.2, Lustre 2.5.1
-
3
-
11237
Description
When volatile files are created, a CREATE changelog record is emitted. But no record says that file was also immediately unlinked.
Either we need a UNLINK record for that FID, or a special flag in CLOSE record.
By the way, I do not think lustre_rsync is handling volatile files, but more especially it should handled LAYOUT records rather than volatile files.
Attachments
Issue Links
- is related to
-
-
- Resolved
-
Activity
I have been able to confirm that this issue is linked to the fact that volatile file usage is causing MDT objects orphans/leak. I will (finaly/hopefully?) have time now to push a patch to fix ...
Hello Aurelien, I am back on this now, and I will check the issue you raised about "no CL events for open-unlink files, even it they are not volatile ones", and let you know soon.
even if generated, the CL records for the Volatile are useless, but am I right ?
It depends what you want to track. So far, there is no important feature requiring them. But you can imagine several use cases where dropping them could be an issue. Especially on different accounting cases. This offers a way to create stealth files and uses disk space without any changelog speaking of it.
no because a volatile file is a std unlinked file
It could be detected as it has a special name, but this would be ugly.
If not, I will be back with the "volatile flag" proposal, and this for all CL records/events ?
Currently the issue is more than some events are raised but not others. In this specific cases no REMOVE events for VOLATILE file. It seems the issue is no CL events for open-unlink files, even it they are not volatile ones. To be checked but maybe this is the real bug.
This is something important IMHO, but it does not need to be a blocker for 2.5.1
> even if generated, the CL records for the Volatile are useless, but am I right ?
yes today but may be in the future
>If yes, I wonder if it could be possible to easily detect them from the CL layer/routines and simply avoid them all ?
no because a volatile file is a std unlinked file
>If not, I will be back with the "volatile flag" proposal, and this for all CL records/events ?
I think this is the right solution
Ok, I am VERY late on this ticket, and by re-reading it, I understand that even if generated, the CL records for the Volatile are useless, but am I right ?
If yes, I wonder if it could be possible to easily detect them from the CL layer/routines and simply avoid them all ?
If not, I will be back with the "volatile flag" proposal, and this for all CL records/events ?
After a volatile is created, many action like layout swap, or access by FID will involve the volatile FID. So removing all the CL will be difficult. IMO we have to keep the CREATE event, with a Volatile flag or followed by an unlink event. The issue with the second is that the unlink is fake, in particular the directory mtime will not change. So an unlink event with a volatile flag ?
Sorry I am late on this, but if I fully understand, we just want to avoid any ChangeLog record for volatile files ??…
Please note that all other Changelog record are also raised when dealing with a volatile file. If you do SETATTR or change HSM flags, etc... this will raise also records for the volatile FID. If we just remove CREATE record, there will be other records for a FID that has never been created, from a CL point of view
Indeed, let's not add CREATE changelog for volatile files.
I've reopened
LU-3696to track the fix for MDT object leak for volatile files separately. That problem is critical since it is leaking space in the filesystem (both on the MDT and OST) for every volatile file used and needs to be back-ported to 2.4.x and 2.5.x. The presence/absence of a changelog record for volatile files is a much more complex issue and I think it can be fixed and landed separately.