[LU-8856] ZFS-MDT 100% full. Cannot delete files.

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.4
• Affects Version/s: Lustre 2.8.0
• Labels: llnl
• Environment: CentOS 6.8 2.6.32_504.30.3.el6.x86_64, Lustre 2.8.0 (g0bcd520), ZFS 0.6.5.4-1

Epic/Theme

zfs

• Severity: 2

Epic

metadata zfs

• Assignee: Alex Zhuravlev
• Reporter: Jeff Johnson (Inactive)

Description

End Customer: MSU (Michigan State Univ)

A user generated tons of small files and exhausted the available inodes of the MDT (single MDT, no DNE). Any attempts at deleting files as root fail.

I looked at LU-8787 and LU-8714 but they don't seem to follow this closely enough.

zdb -d ls15-mds-00.mdt/mdt
Dataset ls15-mds-00.mdt/mdt [ZPL], ID 66, cr_txg 20442, 2.82T, 280362968 objects

ls15-mds-00.mdt/mdt 2.82T 0 2.82T /ls15-mds-00.mdt/mdt

[root@lac-373 roth]# lfs df -i
UUID Inodes IUsed IFree IUse% Mounted on
ls15-MDT0000_UUID 280362968 280362968 0 100% /mnt/ls15[MDT:0]

But we can't remove any files:

[root@lac-000 1mk5_5998]# rm tor.mat
rm: cannot remove `tor.mat': No space left on device

I'm going to take a stab at deregistering the changelog which might free up enough space to get the MDT able to process some file deletions. If anyone has any other 'best practices' please advise.

Issue Links

• is related to: LU-10732 sanity-lfsck test_9a: FAIL: (7) Failed to get expected 'completed'
• is related to: LU-7340 ChangeLogs catalog full condition should be handled more gracefully

Comments

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/31751/
Subject: LU-8856 osd: mark specific transactions netfree
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: a8c7a32fd7fc54e9717e23f208b40c8ff93b81e4

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31751
Subject: LU-8856 osd: mark specific transactions netfree
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: aaafe65b47925dd83b2138d55843ac61af2967a8

[Peter Jones]

Landed for 2.11

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/31444/
Subject: LU-8856 osd: mark specific transactions netfree
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 106abc184d8b57de560dc1874683ce5487dcf30a

[Gerrit Updater]

Alex Zhuravlev ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31444
Subject: LU-8856 osd: mark specific transactions netfree
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8bc89e5b3aeda1bf15f2ff6fc53651e470c0a6c6

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31443
Subject: LU-8856 osd: mark specific transactions netfree
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: d47e3892f83d2d1bdb21653381a8d1ec0db68a4a

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/31442/
Subject: Revert "LU-8856 osd: mark specific transactions netfree"
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c2caa40bd38e7645dc4ac90552e12e3fb7fde476

[Peter Jones]

Reopening due to LU-10732

[Gerrit Updater]

Oleg Drokin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31442
Subject: Revert "LU-8856 osd: mark specific transactions netfree"
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 83be173e2848e3b81d6fb2123d70d0cf614105a8

[Peter Jones]

Landed for 2.11