[LU-11638] lfs fid2path should list open-unlinked files

• Type: Improvement
• Resolution: Unresolved
• Priority: Major
• Affects Version/s: Lustre 2.10.5
• Labels: debug medium usability
• Severity: 3
• Assignee: Andreas Dilger
• Reporter: Mahmoud Hanafi

Description

We delete a large file 15TB (4 stripes), but the objects are not deleting.

pfe24 ~ # lfs fid2path /nobackupp8 0x360635236:0xbd:0x0
fid2path: error on FID 0x360635236:0xbd:0x0: No such file or directory

debugfs:  stat <946398>
Inode: 946398   Type: regular    Mode:  0666   Flags: 0x80000
Generation: 1914950395    Version: 0x00000030:005b5f49
User: 12137   Group: 21205   Project:     0   Size: 3857685938176
File ACL: 0
Links: 1   Blockcount: 7534584632
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x5be3096c:00000000 -- Wed Nov  7 07:49:00 2018
 atime: 0x5be1ae0c:00000000 -- Tue Nov  6 07:06:52 2018
 mtime: 0x5be3096c:00000000 -- Wed Nov  7 07:49:00 2018
crtime: 0x5be1ace3:1d2d1844 -- Tue Nov  6 07:01:55 2018
Size of extra inode fields: 32
Extended attributes:
  trusted.lma (64)
  lma: fid=[0x100000000:0xa104e0f:0x0] compat=18 incompat=0
  fid: parent=[0x360635236:0xbd:0x0] stripe=3 stripe_size=1048576 stripe_count=4
EXTENTS:
(ETB0):234918342, (ETB1):234918343, (ETB2):234918298, (0-2047):3853318144-3853320191, (2048-4095):3854127104-3854129151, (4096-6143):3854174208-3854176255, (6144-8191):3854239744-3854241791, (8192-10239):3854297088-3854299135, (10240-12287):3854319616-3854321663, (12288-14335):3854405632-3854407679, (14336-16383):3854417920-38544199
67, (16384-18431):3854473216-3854475263, (18432-20479):3854772224-3854774271, (20480-22527):3854839808-3854841855, (22528-24575):3854944256-3854946303, (24576-26623):3855372288-3855374335, (26624-28671):3855370240-3855372287, (28672-30719):3855405056-3855407103, (30720-34815):3855431680-3855435775, (34816-36863):3855423488-385542553
5, (36864-38911):3855654912-3855656959, (38912-40959):3855663104-3855665151, (40960-43007):3855669248-3855671295, (43008-45055):3855673344-3855675391, (45056-47103):3855691776-3855693823, (47104-49151):3855702016-3855704063, (49152-53247):3855718400-3855722495, (53248-55295):3855730688-3855732735, (55296-57343):3855753216-3855755263
, (57344-59391):3855747072-3855749119, (59392-61439):3855765504-3855767551, (61440-65535):3855783936-3855788031, (65536-67583):3854090240-3854092287, (67584-69631):3854106624-3854108671, (69632-71679):3854204928-3854206975, (71680-73727):3854295040-3854297087, (73728-75775):3854374912-3854376959, (75776-77823):3854413824-3854415871,
 (77824-79871):3854419968-3854422015, (79872-81919):3854712832-3854714879, (81920-83967):3854811136-3854813183, (83968-86015):3854845952-3854847999, (86016-88063):3854950400-3854952447, (88064-90111):3855187968-3855190015, (90112-92159):3855368192-3855370239, (92160-94207):3855403008-3855405055, (94208-96255):3855409152-3855411199, 
(96256-98303):3855417344-3855419391, (98304-100351):3855448064-3855450111, (100352-102399):3855650816-3855652863, (102400-104447):3855665152-3855667199, (104448-106495):3855683584-3855685631, (106496-108543):3855689728-3855691775, (108544-110591):3855708160-3855710207, (110592-112639):3855714304-3855716351, (112640-114687):385572659
2-3855728639, (114688-116735):3855749120-3855751167, (116736-118783):3855773696-3855775743, (118784-120831):3855767552-3855769599, (120832-124927):3855788032-3855792127, (124928-126975):3855818752-3855820799, (126976-129023):3855831040-3855833087, (129024-131071):3855855616-3855857663, (131072-133119):3853316096-3853318143, (133120-
135167):3854098432-3854100479, (135168-137215):3854112768-3854114815, (137216-139263):3854178304-3854180351, (139264-141311):3854266368-3854268415, (141312-143359):3854292992-3854295039, (143360-145407):3854315520-3854317567, (145408-147455):3854428160-3854430207, (147456-149503):3854409728-3854411775, (149504-151551):3854452736-385
4454783, (151552-153599):3854471168-3854473215, (153600-155647):3854755840-3854757887, (155648-157695):3854835712-3854837759, (157696-159743):3854942208-3854944255, (159744-161791):3855101952-3855103999, (161792-163839):3855355904-3855357951, (163840-165887):3855413248-3855415295, (165888-167935):3855441920-3855443967, (167936-16998
3):3855421440-3855423487, (169984-172031):3855642624-3855644671, (172032-174079):3855661056-3855663103, (174080-176127):3855671296-3855673343, (176128-180223):3855677440-3855681535, (180224-182271):3855695872-3855697919, (182272-184319):3855699968-3855702015, (184320-188415):3855704064-3855708159, (188416-190463):3855710208-38557122
55, (190464-194559):3855736832-3855740927, (194560-196607):3855728640-3855730687, (196608-198655):3854104576-3854106623, (198656-200703):3854143488-3854145535, (200704-202751):3854192640-3854194687, (202752-204799):3854264320-3854266367, (204800-206847):3854299136-3854301183, (206848-208895):3854385152-3854387199, (208896-210943):38
54407680-3854409727, (210944-212991):3854411776-3854413823, (212992-215039):3854454784-3854456831, (215040-217087):3854747648-3854749695, (217088-219135):3854813184-3854815231, (219136-221183):3854850048-3854852095, (221184-223231):3854952448-3854954495, (223232-225279):3855351808-3855353855, (225280-227327):3855357952-3855359999, (
227328-229375):3855396864-3855398911, (229376-231423):3855415296-3855417343, (231424-233471):3855435776-3855437823, (233472-235519):3855648768-3855650815, (235520-237567):3855659008-3855661055, (237568-239615):3855667200-3855669247, (239616-241663):3855675392-3855677439, (241664-243711):3855681536-3855683583, (243712-245759):3855687
680-3855689727, (245760-247807):3855740928-3855742975, (247808-251903):3855722496-3855726591, (251904-253951):3855734784-3855736831, (253952-255999):3855769600-3855771647, (256000-258047):3855745024-3855747071, (258048-260095):3855759360-3855761407, (260096-262143):3855777792-3855779839, (262144-264191):3855761408-3855763455, (26419
2-266239):3855757312-3855759359, (266240-270335):3855800320-3855804415, (270336-274431):3855792128-3855796223, (274432-276479):3855835136-3855837183, (276480-278527):3855865856-3855867903, (278528-280575):3855872000-3855874047, (280576-282623):3855880192-3855882239, (282624-284671):3855892480-3855894527, (284672-286719):3855904768-3
855906815, (286720-288767):3855910912-3855912959, (288768-290815):3855919104-3855921151, (290816-292863):3855951872-3855953919, (292864-294911):3855947776-3855949823, (294912-296959):3855960064-3855962111, (296960-299007):3856050176-3856052223, (299008-301055):3856064512-3856066559, (301056-303103):3856101376-3856103423, (303104-305
151):3856996352-3856998399, (305152-307199):3857031168-3857033215, (307200-309247):3857041408-3857043455, (309248-311295):3857084416-3857086463, (311296-313343):3857057792-3857059839, (313344-315391):3857078272-3857080319, (315392-317439):3857102848-3857104895, (317440-319487):3857123328-3857125375, (319488-321535):3857129472-385713
1519, (321536-323583):3857145856-3857147903, (323584-325631):3857184768-3857186815, (325632-327679):3857258496-3857260543, (327680-331775):3855804416-3855808511, (331776-333823):3855812608-3855814655, (333824-335871):3855828992-3855831039, (335872-337919):3855833088-3855835135, (337920-339967):3855869952-3855871999, (339968-342015):
3855878144-3855880191, (342016-344063):3855902720-3855904767, (344064-348159):3855906816-3855910911, (348160-350207):3855927296-3855929343, (350208-352255):3855953920-3855955967, (352256-354303):3855945728-3855947775, (354304-356351):3855962112-3855964159, (356352-360447):3855984640-3855988735, (360448-362495):3856005120-3856007167,
 (362496-366591):3856041984-3856046079, (366592-368639):3856056320-3856058367, (368640-370687):3856066560-3856068607, (370688-372735):3856459776-3856461823, (372736-374783):3856998400-3857000447, (374784-376831):3857047552-3857049599, (376832-380927):3857074176-3857078271, (380928-382975):3857096704-3857098751, (382976-385023):38571
41760-3857143807, (385024-387071):3857158144-3857160191, (387072-389119):3857260544-3857262591, (389120-391167):3857270784-3857272831, (391168-393215):3857285120-3857287167, (393216-397311):3855808512-3855812607, (397312-399359):3855796224-3855798271, (399360-401407):3855839232-3855841279, (401408-403455):3855837184-3855839231, (403
456-405503):3855867904-3855869951, (405504-409599):3855874048-3855878143, (409600-411647):3855898624-3855900671, (411648-413695):3855890432-3855892479, (413696-415743):3855937536-3855939583, (415744-417791):3855925248-3855927295, (417792-419839):3855921152-3855923199, (419840-421887):3855941632-3855943679, (421888-423935):3855970304
-3855972351, (423936-425983):3855964160-3855966207, (425984-428031):3856054272-3856056319, (428032-430079):3856058368-3856060415, (430080-432127):3856080896-3856082943, (432128-434175):3856099328-3856101375, (434176-436223):3857004544-3857006591, (436224-438271):3857029120-3857031167, (438272-440319):3857049600-3857051647, (440320-4
42367):3857039360-3857041407, (442368-446463):3857061888-3857065983, (446464-448511):3857080320-3857082367, (448512-450559):3857106944-3857108991, (450560-452607):3857125376-3857127423, (452608-454655):3857139712-3857141759, (454656-456703):3857217536-3857219583, (456704-458751):3857225728-3857227775, (458752-462847):3855886336-3855
890431, (462848-464895):3855931392-3855933439, (464896-466943):3855917056-3855919103, (466944-468991):3855923200-3855925247, (468992-471039):3855966208-3855968255, (471040-473087):3855958016-3855960063, (473088-475135):3855980544-3855982591, (475136-477183):3856068608-3856070655, (477184-479231):3856072704-3856074751, (479232-481279
):3856093184-3856095231, (481280-483327):3856457728-3856459775, (483328-485375):3856967680-3856969727, (485376-487423):3857014784-3857016831, (487424-489471):3857037312-3857039359, (489472-493567):3857043456-3857047551, (493568-495615):3857055744-3857057791, (495616-497663):3857065984-3857068031, (497664-499711):3857082368-385708441
5, (499712-501759):3857115136-3857117183, (501760-503807):3857133568-3857135615, (503808-505855):3857137664-3857139711, (505856-507903):3857242112-3857244159, (507904-509951):3857272832-3857274879, (509952-511999):3857303552-3857305599, (512000-514047):3857287168-3857289215, (514048-516095):3857336320-3857338367, (516096-518143):385
7573888-3857575935, (518144-520191):3857790976-3857793023, (520192-522239):3857823744-3857825791, (522240-524287):3857944576-3857946623, (524288-526335):3857268736-3857270783, (526336-528383):3857305600-3857307647, (528384-530431):3857319936-3857321983, (530432-532479):3857330176-3857332223, (532480-534527):3857762304-3857764351, (5
34528-536575):3857807360-3857809407, (536576-538623):3857811456-3857813503, (538624-540671):3857936384-3857938431, (540672-542719):3857981440-3857983487, (542720-544767):3857985536-3857987583, (544768-546815):3857991680-3857993727, (546816-550911):3858003968-3858008063, (550912-552959):3858032640-3858034687, (552960-555007):38580654
08-3858067455, (555008-557055):3858079744-3858081791, (557056-559103):3858114560-3858116607, (559104-561151):3858210816-3858212863, (561152-563199):3858214912-3858216959, (563200-565247):3858247680-3858249727, (565248-567295):3858325504-3858327551, (567296-571391):3858335744-3858339839, (571392-573439):3858427904-3858429951, (573440
-575487):3858497536-3858499583, (575488-577535):3858550784-3858552831, (577536-579583):3858593792-3858595839, (579584-581631):3858575360-3858577407, (581632-583679):3858599936-3858601983, (583680-585727):3858622464-3858624511, (585728-587775):3858632704-3858634751, (587776-589823):3858642944-3858644991, (589824-591871):3857326080-38
57328127, (591872-593919):3857367040-3857369087, (593920-595967):3857575936-3857577983, (595968-598015):3857797120-3857799167, (598016-600063):3857809408-3857811455, (600064-602111):3857934336-3857936383, (602112-604159):3857946624-3857948671, (604160-606207):3857975296-3857977343, (606208-608255):3858001920-3858003967, (608256-6103
03):3858022400-3858024447, (610304-612351):3858067456-3858069503, (612352-614399):3858075648-3858077695, (614400-616447):3858087936-3858089983, (616448-618495):3858100224-3858102271, (618496-620543):3858206720-3858208767, (620544-622591):3858204672-3858206719, (622592-624639):3858229248-3858231295, (624640-626687):3858302976-3858305
023, (626688-628735):3858339840-3858341887, (628736-630783):3858370560-3858372607, (630784-634879):3858528256-3858532351, (634880-636927):3858540544-3858542591, (636928-638975):3858573312-3858575359, (638976-641023):3858577408-3858579455, (641024-643071):3858626560-3858628607, (643072-645119):3858630656-3858632703, (645120-647167):3
858657280-3858659327, (647168-651263):3858647040-3858651135, (651264-653311):3858704384-3858706431, (653312-655359):3858794496-3858796543, (655360-657407):3857340416-3857342463, (657408-659455):3857551360-3857553407, (659456-661503):3857752064-3857754111, (661504-663551):3857803264-3857805311, (663552-665599):3857854464-3857856511, 
(665600-667647):3857950720-3857952767, (667648-669695):3857973248-3857975295, (669696-671743):3857987584-3857989631, (671744-673791):3858008064-3858010111, (673792-675839):3858024448-3858026495, (675840-677887):3858069504-3858071551, (677888-679935):3858083840-3858085887, (679936-681983):3858102272-3858104319, (681984-684031):385820
2624-3858204671, (684032-686079):3858216960-3858219007, (686080-688127):3858286592-3858288639, (688128-690175):3858321408-3858323455, (690176-692223):3858341888-3858343935, (692224-694271):3858358272-3858360319, (694272-696319):3858354176-3858356223, (696320-698367):3858466816-3858468863, (698368-700415):3858544640-3858546687, (7004
16-702463):3858532352-3858534399, (702464-704511):3858581504-3858583551, (704512-706559):3858610176-3858612223, (706560-708607):3858614272-3858616319, (708608-710655):3858638848-3858640895, (710656-714751):3858659328-3858663423, (714752-716799):3858685952-3858687999, (716800-718847):3858681856-3858683903, (718848-720895):3858710528-
3858712575, (ETB2):3858664192, (720896-722943):3857989632-3857991679, (722944-724991):3858020352-3858022399, (724992-727039):3858057216-3858059263, (727040-729087):3858071552-3858073599, (729088-731135):3858096128-3858098175, (731136-733183):3858104320-3858106367, (733184-737279):3858225152-3858229247, (737280-739327):3858305024-385
8307071, (739328-741375):3858352128-3858354175, (741376-743423):3858356224-3858358271, (743424-745471):3858495488-3858497535, (745472-749567):3858536448-3858540543, (749568-751615):3858579456-3858581503, (751616-753663):3858571264-3858573311, (753664-755711):3858606080-3858608127, (755712-757759):3858612224-3858614271, (757760-75980
7):3858640896-3858642943, (759808-761855):3858644992-3858647039, (ETB2):3858629888, (761856-763903):3858667520-3858669567, (763904-765951):3858679808-3858681855, (765952-767999):3858706432-3858708479, (768000-770047):3858651136-3858653183, (770048-772095):3858735104-3858737151, (772096-774143):3858825216-3858827263, (774144-776191):
3858849792-3858851839, (776192-778239):3858857984-3858860031, (778240-782335):3858864128-3858868223, (782336-784383):3858872320-3858874367, (784384-786431):3858907136-3858909183, (786432-788479):3858677760-3858679807, (788480-790527):3858683904-3858685951, (790528-792575):3858702336-3858704383, (792576-794623):3858796544-3858798591,
 (794624-796671):3858847744-3858849791, (796672-798719):3858851840-3858853887, (798720-800767):3858886656-3858888703, (800768-802815):3858882560-3858884607, (802816-804863):3858870272-3858872319, (804864-806911):3858900992-3858903039, (806912-808959):3858905088-3858907135, (808960-811007):3858972672-3858974719, (811008-813055):38589
89056-3858991103, (813056-815103):3858997248-3858999295, (815104-817151):3859013632-3859015679, (817152-819199):3859036160-3859038207, (819200-821247):3859179520-3859181567, (821248-823295):3859191808-3859193855, (823296-825343):3859212288-3859214335, (825344-827391):3859230720-3859232767, (827392-829439):3859271680-3859273727, (829
440-831487):3859265536-3859267583, (831488-833535):3859283968-3859286015, (833536-835583):3859363840-3859365887, (835584-837631):3859560448-3859562495, (837632-839679):3859599360-3859601407, (839680-841727):3859623936-3859625983, (841728-843775):3859654656-3859656703, (843776-845823):3859695616-3859697663, (845824-847871):3859750912
-3859752959, (847872-851967):3859781632-3859785727, (851968-854015):3858743296-3858745343, (854016-856063):3858833408-3858835455, (856064-858111):3858855936-3858857983, (858112-860159):3858862080-3858864127, (860160-862207):3858876416-3858878463, (862208-864255):3858868224-3858870271, (864256-866303):3858894848-3858896895, (866304-8
68351):3858903040-3858905087, (868352-870399):3858941952-3858943999, (870400-872447):3858991104-3858993151, (872448-874495):3858999296-3859001343, (874496-884735):3859003392-3859013631, (884736-886783):3859019776-3859021823, (886784-888831):3859204096-3859206143, (888832-890879):3859208192-3859210239, (890880-892927):3859238912-3859
240959, (892928-894975):3859261440-3859263487, (894976-897023):3859306496-3859308543, (897024-899071):3859286016-3859288063, (899072-901119):3859353600-3859355647, (901120-903167):3859582976-3859585023, (903168-905215):3859632128-3859634175, (905216-909311):3859666944-3859671039, (909312-911359):3859660800-3859662847, (911360-913407
):3859685376-3859687423, (913408-915455):3859681280-3859683327, (915456-917503):3859742720-3859744767, (917504-919551):3858874368-3858876415, (919552-921599):3858884608-3858886655, (921600-923647):3858892800-3858894847, (923648-925695):3858933760-3858935807, (925696-927743):3858993152-3858995199, (927744-929791):3859001344-385900339
1, (929792-931839):3859015680-3859017727, (931840-935935):3859023872-3859027967, (935936-937983):3859038208-3859040255, (937984-940031):3859189760-3859191807, (940032-942079):3859210240-3859212287, (942080-944127):3859275776-3859277823, (944128-946175):3859273728-3859275775, (946176-948223):3859288064-3859290111, (948224-950271):385
9402752-3859404799, (950272-952319):3859570688-3859572735, (952320-954367):3859597312-3859599359, (954368-956415):3859601408-3859603455, (956416-958463):3859636224-3859638271, (958464-960511):3859662848-3859664895, (960512-962559):3859683328-3859685375, (962560-964607):3859699712-3859701759, (964608-966655):3859748864-3859750911, (9
66656-970751):3859789824-3859793919, (970752-972799):3859798016-3859800063, (972800-974847):3859828736-3859830783, (974848-978943):3859808256-3859812351, (978944-980991):3859855360-3859857407, (980992-983039):3859884032-3859886079, (983040-985087):3858978816-3858980863, (985088-987135):3859017728-3859019775, (987136-989183):38590402
56-3859042303, (989184-991231):3859206144-3859208191, (991232-995327):3859257344-3859261439, (995328-997375):3859263488-3859265535, (997376-999423):3859290112-3859292159, (999424-1001471):3859453952-3859455999, (1001472-1003519):3859591168-3859593215, (1003520-1005567):3859585024-3859587071, (1005568-1007615):3859634176-3859636223, 
debugfs:  ncheck 946398
Inode   Pathname
946398  /O/0/d15/168840719

 

Issue Links

• is duplicated by: LU-11687 allow "lfs fid2path" on open-unlinked files
• is related to: LU-13527 allow OST FID lookup via lfs fid2path

Comments

[Gerrit Updater]

Joseph Gmitter ([e-mail]) merged in patch https://review.whamcloud.com/33629/
Subject: LU-11638 admin: debug space management/missing objects
Project: doc/manual
Branch: master
Current Patch Set:
Commit: 5e609ea889c9626dfe558e170d78ab74dda42230

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33629
Subject: LU-11638 admin: debug space management/missing objects
Project: doc/manual
Branch: master
Current Patch Set: 1
Commit: 46989d10fe59fa2df6d38885cf53839da9add6d4

[Andreas Dilger]

Reopening this issue because we don't know what the root cause is. We should be able to handle cleanup of open-unlinked files properly, and/or files that are unlinked from the MDT but held open when the MDS crashed.

I did a small test and saw that lfs fid2path does not report anything for an open-unlinked file:

client$ lfs path2fid /myth/tmp/1M
[0x200003ab4:0x435:0x0]
client$ sleep 10000 < /myth/tmp/1M &
[1] 13806
client$ sudo lfs fid2path /myth [0x200003ab4:0x435:0x0]
/myth/tmp/1M
mds# lctl get_param mdt.*.exports.*.open_files
[e-mail]_files=[0x200003ab4:0x435:0x0]
client$ rm /myth/tmp/1M
mds# lctl get_param mdt.*.exports.*.open_files
[e-mail]_files=[0x200003ab4:0x435:0x0]
client$ sudo lfs fid2path /myth [0x200003ab4:0x435:0x0]
lfs fid2path: cannot find '[0x200003ab4:0x435:0x0]': No such file or directory
client$ lsof /myth
COMMAND   PID   USER   FD   TYPE    DEVICE SIZE/OFF               NODE NAME
sleep   13806 mythtv    0r   REG 35,632494  1048576 144115440203858997 /myth/tmp/1M (deleted)

If this happens again in the future, it would be useful to check the mdt.*.exports.*.open_files list to see if the FID is listed there, which also lists the client NID that is holding it open, and you could use lsof on the client to find and kill the offending process (note that lsof will only report file handles open on the local node).

It would be useful if lfs fid2path listed these open-unlinked files directly, including the client NID and appends (deleted) as lsof does.

[Peter Jones]

Good news - thanks!

[Mahmoud Hanafi]

I was able to delete the object after umounting OST and remount via ldiskfs.

Please close this issue.

[Andreas Dilger]

Officially, no. In theory, yes it is possible to do that (I sometimes do concurrent ldiskfs mounts on my home filesystem for debugging purposes), but it is not something we have officially tested. If you have a quiet time and could "expect" a crash, but are happy if it works without problems or interference with jobs, then I'd say it should probably be OK to do that. Otherwise, I'd suggest to wait until there is a scheduled outage window and try it then. At worst it would crash, but that is not much worse than just unmounting the OST (causing IO to block), remount as ldiskfs and remove the object, then remount the OST and go through recovery.

[Mahmoud Hanafi]

This filesystem was not related to LU-11548. The only that may have happened here is the file was delete while it was still being written to from a different client. And then that write thread was killed.

Is it safe to mount the ost as ldiskfs and delete the object while the OST is mounted as lustre?

[Andreas Dilger]

Is this one of the filesystems related to LU-11548? One of the problems seen in that case was multiple hard links to the same OST object. If that is the case here, it is possible that the "real" object was removed, but a hard linked name was left behind and the space was not freed.

Another possibility is that the object was migrated to a new MDT and the parent FID on the object was not updated. You might be able to search for large files in the directory (possibly belonging to UID 12137 and/or GID 21205) to see if that is the case?

lfs find /nobackupp8 -size +3T -print0 | xargs -0 ls -l

I assume there were not any other events around this time (eg. MDS or OSS crash)? That shouldn't cause an unlinked file to leave orphan objects behind, but knowing the circumstances around this file may help.

The orphan object(s) can be linked to /nobackupp8/.lustre/lost+found/ and/or unlinked by running lctl lfsck_start -A -t layout on the MDS (assuming this is not having other issues at the current time). Alternately, instead of running LFSCK, if the lfs find does not turn up any candidate files, you could mount the OST as type ldiskfs and delete the objects manually.