Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Lustre 2.5.4
-
None
-
RHEL-6.6, lustre-2.5.4
-
2
-
9223372036854775807
Description
We had 4 OSTs that we deactivated because of an imbalance in utilization that was causing ENOSPC messages to our users. We identified a file that was consuming a significant amount of space that we deleted while the OSTs were deactivated. The file is no longer seen in the directory structure (the MDS processed the request), but the objects on the OSTs were not marked as free. After re-activating the OSTs, it doesn't appear that the llog was flushed, which should free up those objects.
At this time, some users are not able to run jobs because they cannot allocated any space.
We understand how this is supposed to work, but as the user in LU-4295 pointed out, it is not.
Please advise.
Attachments
Issue Links
Activity
Mike, is this needed for 2.7.x or only 2.5.x? It would be great to link this to a specific patch/bug that fixed the problem for master if possible.
Finishing testing your patch and it appears to have resolved our issues.
That is interesting that there are no obvious changes between master and b2_5 related to this behavior. Meanwhile I've made a simple fix for this issue, it works for me. Please check it.
Mike Pershin (mike.pershin@intel.com) uploaded a new patch: http://review.whamcloud.com/16612
Subject: LU-7012 osp: don't use OSP when import is deactivated
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: b7daf6b218a34c18330ff6f5d8e023e48bee1e0b
I can reproduce this with 2.5, considering that master is working fine I am going to find out related changes in it and port them to the 2.5
Yu Jian, you are right, I have used master branch instead of b2_5, my mistake. I am repeating local tests with 2.5 now
Hi Mike,
What Lustre version did you test on? Is it Lustre 2.5.x or master branch?
Well, I was trying to reproduce that locally and objects are not deleted while OSP is deactivated but they are deleted immediately when I re-activate OSP back. I used 'lctl --device <osp device> deactivate' command to deactivate an OSP. Then destroy big file that was previously created on that OST. The 'df' shows that space on related OST is not freed, after that I re-activated OSP back and 'df' shows space is returned back. Any thoughts what else may affect that?
This is OSP problem it seems, which doesn't restart llog processing from the point where OST was de-activated. I am testing it locally now.
Mike, there are two separate problems:
1) the current method for doing OST space balancing is to deactivate the OSP and then migrate files (or let users do this gradually), so the deactivated OST will not be used for new objects. However, deactivating the OSP also prevents the MDS from destroying the objects of unlinked files (since 2.4) so space is never released on the OST, which confuses users. This issue will be addressed by LU-4825 by adding a new method for disabling object allocation on an OST without fully deactivating the OSP, so that the MDS can still process object destroys.
2) when the deactivated OSP is reactivated again, even after restarting the OST, it does not process the unlink llogs (and presumably Astarte logs, but that is harder to check) until the MDS is stopped and restarted. The MDS should begin processing the recovery llogs after the OSP has been reactivated. That is what this bug is for.
Even though LU-4825 will reduce the times when an OSP needs to be deactivated (i.e. Not for space balancing anymore), there are other times when this still needs to be done (e.g. OST offline for maintenance or similar) so recovery llog processing still needs to work.
Andreas, I see the same problem in 2.7 but it works somehow, I suppose that DNE changes fixed that indirectly by adding more synchronization mechanisms in OSP. Meanwhile, I'd add this patch to the 2.7 just as direct fix for that particular problem