[LU-4640] Last unlink should trigger HSM remove by default

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.10.0
• Affects Version/s: Lustre 2.5.0
• Labels: HSM patch
• Severity: 3
• Assignee: Bruno Faccini (Inactive)
• Reporter: Aurelien Degremont (Inactive)

Description

In current code, when an archived file is removed from the file system, no HSM request is triggered. We rely on Changelogs and RobinHood reading them to have it sending corresponding hsm_remove requests to clean those orphans in HSM backend. This is done in purpose, to provide a way to implement "soft unlink". RobinHood will remove the file in the backend after a grace time. During this time, Admins could restore the file from the HSM if they want to (using import feature).

Requiring a RobinHood setup to handle this cleaning is a too big limitation. We should consider modifying this behaviour.

By default, Lustre should automatically add a HSM_REMOVE request for any last unlink. This way, no file will be leaked in the archive.
A tunable should be added to disable this behaviour (should we add this to hsm_policy?) and go back to a mode where an external component is responsible for tracking UNLINK changelogs and add hsm_remove requests when needed (Robinhood) (current behaviour).

Issue Links

• is related to: LU-9654 fix problem of Remove Archive on Last Unlink policy
• is related to: LU-7881 sanity-hsm test_26b: @@@@@@ FAIL: Copytool should have stopped

Comments

[Li Xi (Inactive)]

Hi Bruno,

Thanks for helping. Yeah, I create a new patch based on your latest patch.

I first improved the test 26c and 26d. And after that, the test results immediately shows that the mdt_attr_get_complex returns -2 because the object is deleted. That means, the MA_HSM and MA_INODE needs to be read before unlink happens. That is why I split mdt_handle_last_unlink to mdt_handle_last_unlink_prepare and mdt_handle_last_unlink_commit.

Let's see what is the test result thi time.

[Bruno Faccini (Inactive)]

Ok will try to work again on this patch if there is an urgent need now.
The reason my original patch did not work when being merged was due to some other changes (in test framework and also causing earlier object removal) that had been merged between its final+successful testing/review and the time it has landed.

LU-7881 has addressed the necessary changes required for new sanity-hsm/test_26[a,b,c,d] to comply with new test framework changes.

But I had still work to be done in order to address new condition caused by early object removal.
Just pushed a new patch-set #7 for change #18946 to try fixing this now.

By the way Li, why did you create a new #26980 Change-Id instead working on top of mine/existing ??

[Peter Jones]

Li Xi

It is feasible that we could land a fix for this change earliy in the 2.11 dev cycle

Peter

[Gerrit Updater]

Li Xi ([e-mail]) uploaded a new patch: https://review.whamcloud.com/26980
Subject: LU-4640 mdt: implement Remove Archive on Last Unlink policy
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4d76c26ec74c3b0748c6b0a3bf10ff1eebfab216

[Li Xi (Inactive)]

Hi, Any plan to merge this patch in the near future? I am asking because I realize this is a critical problem for the policy engine that I am working on (Lustre Integrated Policy Engine). This policy engine doesn't require Lustre Changelog, thus the policy engine can not be notified by the Changelog about the unlinking of files.

[Gerrit Updater]

Faccini Bruno ([e-mail]) uploaded a new patch: http://review.whamcloud.com/18946
Subject: LU-4640 mdt: implement Remove Archive on Last Unlink policy
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2ed64a4994a866ce653f10af1c110abe6d506ecc

[Bruno Faccini (Inactive)]

Basically and after more thinking about this, seems that the best way is to revert the revert and then to add additional changes (LU-7881). This should lead to a new commit for the almos the same changes.

[Bruno Faccini (Inactive)]

Humm, but what is the best way to do so? I have tried to first rebase my patch on top of master already containing it and its revert, but nothing/no-change happen.
This could be side effect of the revert that does not really undo the first merge but only applies a patch to revert the original data changes ...
So, do I need to create a new patch by manually back-porting the changes of original patch on top of master ??

[Bruno Faccini (Inactive)]

Well the problem comes from the time that this patch has waited to be landed, when in the meantime patch for LU-7136 was, and thus changed sanity-hsm framework causing this issue ...

Will push a new version/patch-set of http://review.whamcloud.com/14384/ that will integrate my additional change for LU-7881 (http://review.whamcloud.com/#/c/18919/, "use new functions to kill and verify CT death" ).

[Oleg Drokin]

This patch was reverted due to testing problems discussed in LU-7881 causing big disruption to other patches.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/18925/
Subject: Revert "LU-4640 mdt: implement Remove Archive on Last Unlink policy"
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d9f95aa201341d972eeb610471e3c45f1ba12202