[LU-7012] files not being deleted from OST after being re-activated

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.5.4
• Environment: RHEL-6.6, lustre-2.5.4
• Severity: 2
• Assignee: Mikhail Pershin
• Reporter: Dustin Leverman

Description

We had 4 OSTs that we deactivated because of an imbalance in utilization that was causing ENOSPC messages to our users. We identified a file that was consuming a significant amount of space that we deleted while the OSTs were deactivated. The file is no longer seen in the directory structure (the MDS processed the request), but the objects on the OSTs were not marked as free. After re-activating the OSTs, it doesn't appear that the llog was flushed, which should free up those objects.

At this time, some users are not able to run jobs because they cannot allocated any space.

We understand how this is supposed to work, but as the user in LU-4295 pointed out, it is not.

Please advise.

Issue Links

• is duplicated by: LU-4295 removing files on deactivated OST doesn't free up space
• is related to: LU-4825 lfs migrate not freeing space on OST

Comments

[Joseph Gmitter (Inactive)]

Landed for 2.8

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/16937/
Subject: LU-7012 osp: don't use OSP when import is deactivated
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 82cbfd77f33bc33ea047407dfaecf4b04d44930a

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: http://review.whamcloud.com/16937
Subject: LU-7012 osp: don't use OSP when import is deactivated
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9a0a96518ab32908d381d22a4eccbfaa28cafd1d

[Mikhail Pershin]

Andreas, I see the same problem in 2.7 but it works somehow, I suppose that DNE changes fixed that indirectly by adding more synchronization mechanisms in OSP. Meanwhile, I'd add this patch to the 2.7 just as direct fix for that particular problem

[Andreas Dilger]

Mike, is this needed for 2.7.x or only 2.5.x? It would be great to link this to a specific patch/bug that fixed the problem for master if possible.

[James A Simmons]

Finishing testing your patch and it appears to have resolved our issues.

[Mikhail Pershin]

That is interesting that there are no obvious changes between master and b2_5 related to this behavior. Meanwhile I've made a simple fix for this issue, it works for me. Please check it.

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: http://review.whamcloud.com/16612
Subject: LU-7012 osp: don't use OSP when import is deactivated
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: b7daf6b218a34c18330ff6f5d8e023e48bee1e0b

[Mikhail Pershin]

I can reproduce this with 2.5, considering that master is working fine I am going to find out related changes in it and port them to the 2.5

[Mikhail Pershin]

Yu Jian, you are right, I have used master branch instead of b2_5, my mistake. I am repeating local tests with 2.5 now