[LU-4640] Last unlink should trigger HSM remove by default

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.10.0
• Affects Version/s: Lustre 2.5.0
• Labels: HSM patch
• Severity: 3
• Assignee: Bruno Faccini (Inactive)
• Reporter: Aurelien Degremont (Inactive)

Description

In current code, when an archived file is removed from the file system, no HSM request is triggered. We rely on Changelogs and RobinHood reading them to have it sending corresponding hsm_remove requests to clean those orphans in HSM backend. This is done in purpose, to provide a way to implement "soft unlink". RobinHood will remove the file in the backend after a grace time. During this time, Admins could restore the file from the HSM if they want to (using import feature).

Requiring a RobinHood setup to handle this cleaning is a too big limitation. We should consider modifying this behaviour.

By default, Lustre should automatically add a HSM_REMOVE request for any last unlink. This way, no file will be leaked in the archive.
A tunable should be added to disable this behaviour (should we add this to hsm_policy?) and go back to a mode where an external component is responsible for tracking UNLINK changelogs and add hsm_remove requests when needed (Robinhood) (current behaviour).

Issue Links

• is related to: LU-9654 fix problem of Remove Archive on Last Unlink policy
• is related to: LU-7881 sanity-hsm test_26b: @@@@@@ FAIL: Copytool should have stopped

Comments

[Bruno Faccini (Inactive)]

Well the problem comes from the time that this patch has waited to be landed, when in the meantime patch for LU-7136 was, and thus changed sanity-hsm framework causing this issue ...

Will push a new version/patch-set of http://review.whamcloud.com/14384/ that will integrate my additional change for LU-7881 (http://review.whamcloud.com/#/c/18919/, "use new functions to kill and verify CT death" ).

[Oleg Drokin]

This patch was reverted due to testing problems discussed in LU-7881 causing big disruption to other patches.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/18925/
Subject: Revert "LU-4640 mdt: implement Remove Archive on Last Unlink policy"
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d9f95aa201341d972eeb610471e3c45f1ba12202

[Gerrit Updater]

Oleg Drokin ([e-mail]) uploaded a new patch: http://review.whamcloud.com/18925
Subject: Revert "LU-4640 mdt: implement Remove Archive on Last Unlink policy"
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 97b0e7059a9755e918e0760fd21b33186395cf5b

[Bruno Faccini (Inactive)]

Hello Aurelien,
Now my patch to implement RAoLU has finally landed, do you agree that we can close this long standing ticket ??

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/14384/
Subject: LU-4640 mdt: implement Remove Archive on Last Unlink policy
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 74d92933108dc64b110a843352cf3336dca249d0

[Aurelien Degremont (Inactive)]

I agree with you that we do not need to wake up the CDT for each of them. It will be more difficult to write tests however.

[Bruno Faccini (Inactive)]

Now using mdt_agent_record_add() directly raises a new question, should we wake-up (by calling mdt_hsm_cdt_wakeup()) CDT upon each of these hsm_remove requests being created or simply "batch" them in hsm_actions LLOG waiting for CDT to wake-up upon any other event or loop_period timer end ? My feeling is that batching them is a good idea since there is no emergency to handle them ...

[Bruno Faccini (Inactive)]

Patch-set #1 testing has shown that requesting HSM data was unnecessary due to being already done before for a fully unlinked file via

mdt_reint_unlink()->mdo_unlink()->mdd_unlink()->mdd_hsm_archive_exists()

call sequence.
Also mdt_hsm_add_actions() can not be used because it tries to lookup (via mdt_object_find()) object again but hangs/wait then (in lu_object_find[_at](), due to

-EAGAIN

returned for a dying object) since object seems already set as dying via

mdt_reint_unlink()->mdo_unlink()->mdd_unlink()->mdd_finish_unlink()->mdo_destroy()->...

call sequence. I use mdt_agent_record_add() directly instead.

[Gerrit Updater]

Faccini Bruno ([e-mail]) uploaded a new patch: http://review.whamcloud.com/14384
Subject: LU-4640 mdt: implement Remove Archive on Last Unlink policy
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5c285dee336a29c2e1a55835004446fbd89c9c63

[Bruno Faccini (Inactive)]

After more discussions it appears that this new behavior should not be enable by default but upon a new tunable setting.