[LU-4825] lfs migrate not freeing space on OST

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.9.0, Lustre 2.10.0
• Affects Version/s: Lustre 2.4.1
• Labels: ldiskfs
• Environment: SLES 11 SP2 clients, CentOS 6.4 servers (DDN packaged)
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Shawn Hall (Inactive)

Description

We have some OSTs that we let get out of hand and have reached 100% capacity. We have offlined them using "lctl --device <device_num> deactivate" along with others that are approaching capacity. Despite having users delete multi-terabyte files and using the lfs_migrate script (with two patches from LU-4293 included to allow it to use "lfs migrate" as root instead of rsync) to migrate over 100 TB of data (with the full OSTs deactivated), we are not freeing up any space on the OSTs.

Our initial guess was that after the layout swap of the "lfs migrate", the old objects were not being deleted from disk because those OSTs were deactivated on the MDS. Therefore on one OST I re-activated it on the MDS, unmounted from the OSS, and ran an "e2fsck -v -f -p /dev/..." and that seemed to free about 300 GB on the OST. I tried the same procedure on another OST and it did not change anything. The e2fsck output indicates that nothing "happened" in either case.

This is a live, production file system so after yanking two OSTs offline I thought I'd stop testing theories before too many users called

Issue Links

• is related to: LU-11115 OST selection algorithm broken with max_create_count=0 or empty OSTs
• is related to: LU-4295 removing files on deactivated OST doesn't free up space
• is related to: LU-11605 create_count stuck in 0 after changeing max_create_count to 0 and back 20 000
• is related to: LU-8523 sanity test_311: objs not destroyed after unlink
• is related to: LU-7012 files not being deleted from OST after being re-activated
• is related to: LU-5931 Deactivated OST still contains data
• is related to: LUDOC-305 "lctl deactivate/activate" does not work as expected in 19.1. Handling Full OSTs