[LU-5708] Cannot get rid of orphaned objects

• Type: Bug
• Resolution: Incomplete
• Priority: Major
• Affects Version/s: Lustre 2.5.3
• Severity: 3
• Assignee: Dmitry Eremin (Inactive)
• Reporter: Oliver Mangold

Description

After some testing and benchmarking of a fresh filesystem, we found 1.4TB worth of orphaned objects. What we did was mostly several runs of tar (kernel tree extraction) and IOR benchmark.

After deleting all temporary files from these tests, we end up with a Lustre containing just 108k of files:

[root@n0101 ~]# du -sh /mnt/lustre/lnec/
108K /mnt/lustre/lnec/

but according to df, 1..4TB are in use:

[root@n0101 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
...
10.4.0.102@o2ib:10.4.0.101@o2ib:/lnec 175T 1.4T 174T 1% /mnt/lustre/lnec

Mounting one of the osts as ldiskfs and checking the contents we find that it contains 228GB worth of objects:

oss01:ost0# du -sh --total O/*
85G O/0
136K O/1
136K O/10
144G O/2
136K O/200000003
228G total

We have 6 osts in total and all of them are in a comparable state, so we add up to 1.4TB of objects.

Trying 'lctl lfsck_start' on the MDT or OSTs doesn't change that.

Issue Links

• is related to: LU-6414 du and df disagree for used space