[LU-8856] ZFS-MDT 100% full. Cannot delete files.

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.4
• Affects Version/s: Lustre 2.8.0
• Labels: llnl
• Environment: CentOS 6.8 2.6.32_504.30.3.el6.x86_64, Lustre 2.8.0 (g0bcd520), ZFS 0.6.5.4-1

Epic/Theme

zfs

• Severity: 2

Epic

metadata zfs

• Assignee: Alex Zhuravlev
• Reporter: Jeff Johnson (Inactive)

Description

End Customer: MSU (Michigan State Univ)

A user generated tons of small files and exhausted the available inodes of the MDT (single MDT, no DNE). Any attempts at deleting files as root fail.

I looked at LU-8787 and LU-8714 but they don't seem to follow this closely enough.

zdb -d ls15-mds-00.mdt/mdt
Dataset ls15-mds-00.mdt/mdt [ZPL], ID 66, cr_txg 20442, 2.82T, 280362968 objects

ls15-mds-00.mdt/mdt 2.82T 0 2.82T /ls15-mds-00.mdt/mdt

[root@lac-373 roth]# lfs df -i
UUID Inodes IUsed IFree IUse% Mounted on
ls15-MDT0000_UUID 280362968 280362968 0 100% /mnt/ls15[MDT:0]

But we can't remove any files:

[root@lac-000 1mk5_5998]# rm tor.mat
rm: cannot remove `tor.mat': No space left on device

I'm going to take a stab at deregistering the changelog which might free up enough space to get the MDT able to process some file deletions. If anyone has any other 'best practices' please advise.

Issue Links

• is related to: LU-10732 sanity-lfsck test_9a: FAIL: (7) Failed to get expected 'completed'
• is related to: LU-7340 ChangeLogs catalog full condition should be handled more gracefully