[LU-6449] Broadcast HSM remove requests with undetermined archive_id to all registered copytools

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.10.0
• Assignee: Bruno Faccini (Inactive)
• Reporter: Bruno Faccini (Inactive)

Description

The need for this change has been unveiled by configuring RobinHood delayed soft_rm feature, where HSM remove requests are submitted with no/Null archive_id, because RBH current version does not save archive_id infromation in its SOFT_RM DB table.
Since the corresponding files/FIDs have been fully unlinked, their Inodes/xattrs are no longer available in the name-space, and thus CDT is unable to retrieve the archive_id information by himself.
This leads to a situation where, without a copytool being registered to handle all/32 archive_ids, these HSM remove request can not be processed and will leak in archive.

Issue Links

• is related to: LU-6494 hsm: lfs cannot remove an archived file that has been deleted
• is related to: LU-9443 omit changelogs for volatile files

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/21077/
Subject: LU-6449 mdt: correct status and ChangeLogs for dead objects
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3ed1444d3fa4e31fbb99c82a478802c0dbcb168e

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/22212/
Subject: LU-6449 hsm: handle completions with non-existent objects
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4465f1cf50dc16b37a9ea23efec78b17fd1936eb

[Gerrit Updater]

This Gerrit Updater msg has been removed because it had been wrongly added to this ticket due to a typo (LU-6449 vs LU-6499) in the header/1st-line of commit msg.

[Gerrit Updater]

Faccini Bruno ([e-mail]) uploaded a new patch: http://review.whamcloud.com/22212
Subject: LU-6449 hsm: handle completions with non-existent objects
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f597ff217fe1eda6d84d551cd6691bfd0141409d

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/20991/
Subject: LU-6449 mdt: broadcast orphan hsm_remove requests
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: fc84852e990dd3174b63ab8e16e6bc3c0b1b092a

[Gerrit Updater]

Faccini Bruno ([e-mail]) uploaded a new patch: http://review.whamcloud.com/21077
Subject: LU-6449 mdt: correct status and ChangeLogs for dead objects
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4e38194561ec193bef858d32f3824994fb660b18

[Gerrit Updater]

Faccini Bruno ([e-mail]) uploaded a new patch: http://review.whamcloud.com/20991
Subject: LU-6449 mdt: broadcast orphan hsm_remove requests
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5a5aead31f9ebab8219ef47446174294cb95c80b

[Bruno Faccini (Inactive)]

RobinHood side/specific patch is at https://review.gerrithub.io/238708 ("hsm_rm: store archive_id and use it for deferred remove requests").

[Bruno Faccini (Inactive)]

In fact both are complementary! The Lustre change for this ticket will avoid to leak files on archives and I will also push a patch to RBH repo in order to have it store the archive_id.

[Thomas Leibovici]

There are cases where robinhood cannot know where the file has been stored.
For instance, if it processes the following sequence of changelog asynchronously: HSM (archive, rc=0), UNLINK.
In this case, it's too late to query the file to know where it has been stored.

The intent of sending "hsm_remove" request is to clean copies in the archive.
IMO it would make sense that sending a hsm_remove request with no archive_id (0) would clean it in every backend.
It would be helpful for the case I just described.

[Andreas Dilger]

Wouldn't it be better to fix RH to store the archive_id into this SOFT_RM DB table?