https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11638 Lustre <p>We delete a large file 15TB (4 stripes), but the objects are not deleting.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> pfe24 ~ # lfs fid2path /nobackupp8 0x360635236:0xbd:0x0 fid2path: error on FID 0x360635236:0xbd:0x0: No such file or directory </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> debugfs: stat &lt;946398&gt; Inode: 946398 Type: regular Mode: 0666 Flags: 0x80000 Generation: 1914950395 Version: 0x00000030:005b5f49 User: 12137 Group: 21205 Project: 0 Size: 3857685938176 File ACL: 0 Links: 1 Blockcount: 7534584632 Fragment: Address: 0 <span class="code-object">Number</span>: 0 Size: 0 ctime: 0x5be3096c:00000000 -- Wed Nov 7 07:49:00 2018 atime: 0x5be1ae0c:00000000 -- Tue Nov 6 07:06:52 2018 mtime: 0x5be3096c:00000000 -- Wed Nov 7 07:49:00 2018 crtime: 0x5be1ace3:1d2d1844 -- Tue Nov 6 07:01:55 2018 Size of extra inode fields: 32 Extended attributes: trusted.lma (64) lma: fid=[0x100000000:0xa104e0f:0x0] compat=18 incompat=0 fid: parent=[0x360635236:0xbd:0x0] stripe=3 stripe_size=1048576 stripe_count=4 EXTENTS: (ETB0):234918342, (ETB1):234918343, (ETB2):234918298, (0-2047):3853318144-3853320191, (2048-4095):3854127104-3854129151, (4096-6143):3854174208-3854176255, (6144-8191):3854239744-3854241791, (8192-10239):3854297088-3854299135, (10240-12287):3854319616-3854321663, (12288-14335):3854405632-3854407679, (14336-16383):3854417920-38544199 67, (16384-18431):3854473216-3854475263, (18432-20479):3854772224-3854774271, (20480-22527):3854839808-3854841855, (22528-24575):3854944256-3854946303, (24576-26623):3855372288-3855374335, (26624-28671):3855370240-3855372287, (28672-30719):3855405056-3855407103, (30720-34815):3855431680-3855435775, (34816-36863):3855423488-385542553 5, (36864-38911):3855654912-3855656959, (38912-40959):3855663104-3855665151, (40960-43007):3855669248-3855671295, (43008-45055):3855673344-3855675391, (45056-47103):3855691776-3855693823, (47104-49151):3855702016-3855704063, (49152-53247):3855718400-3855722495, (53248-55295):3855730688-3855732735, (55296-57343):3855753216-3855755263 , (57344-59391):3855747072-3855749119, (59392-61439):3855765504-3855767551, (61440-65535):3855783936-3855788031, (65536-67583):3854090240-3854092287, (67584-69631):3854106624-3854108671, (69632-71679):3854204928-3854206975, (71680-73727):3854295040-3854297087, (73728-75775):3854374912-3854376959, (75776-77823):3854413824-3854415871, (77824-79871):3854419968-3854422015, (79872-81919):3854712832-3854714879, (81920-83967):3854811136-3854813183, (83968-86015):3854845952-3854847999, (86016-88063):3854950400-3854952447, (88064-90111):3855187968-3855190015, (90112-92159):3855368192-3855370239, (92160-94207):3855403008-3855405055, (94208-96255):3855409152-3855411199, (96256-98303):3855417344-3855419391, (98304-100351):3855448064-3855450111, (100352-102399):3855650816-3855652863, (102400-104447):3855665152-3855667199, (104448-106495):3855683584-3855685631, (106496-108543):3855689728-3855691775, (108544-110591):3855708160-3855710207, (110592-112639):3855714304-3855716351, (112640-114687):385572659 2-3855728639, (114688-116735):3855749120-3855751167, (116736-118783):3855773696-3855775743, (118784-120831):3855767552-3855769599, (120832-124927):3855788032-3855792127, (124928-126975):3855818752-3855820799, (126976-129023):3855831040-3855833087, (129024-131071):3855855616-3855857663, (131072-133119):3853316096-3853318143, (133120- 135167):3854098432-3854100479, (135168-137215):3854112768-3854114815, (137216-139263):3854178304-3854180351, (139264-141311):3854266368-3854268415, (141312-143359):3854292992-3854295039, (143360-145407):3854315520-3854317567, (145408-147455):3854428160-3854430207, (147456-149503):3854409728-3854411775, (149504-151551):3854452736-385 4454783, (151552-153599):3854471168-3854473215, (153600-155647):3854755840-3854757887, (155648-157695):3854835712-3854837759, (157696-159743):3854942208-3854944255, (159744-161791):3855101952-3855103999, (161792-163839):3855355904-3855357951, (163840-165887):3855413248-3855415295, (165888-167935):3855441920-3855443967, (167936-16998 3):3855421440-3855423487, (169984-172031):3855642624-3855644671, (172032-174079):3855661056-3855663103, (174080-176127):3855671296-3855673343, (176128-180223):3855677440-3855681535, (180224-182271):3855695872-3855697919, (182272-184319):3855699968-3855702015, (184320-188415):3855704064-3855708159, (188416-190463):3855710208-38557122 55, (190464-194559):3855736832-3855740927, (194560-196607):3855728640-3855730687, (196608-198655):3854104576-3854106623, (198656-200703):3854143488-3854145535, (200704-202751):3854192640-3854194687, (202752-204799):3854264320-3854266367, (204800-206847):3854299136-3854301183, (206848-208895):3854385152-3854387199, (208896-210943):38 54407680-3854409727, (210944-212991):3854411776-3854413823, (212992-215039):3854454784-3854456831, (215040-217087):3854747648-3854749695, (217088-219135):3854813184-3854815231, (219136-221183):3854850048-3854852095, (221184-223231):3854952448-3854954495, (223232-225279):3855351808-3855353855, (225280-227327):3855357952-3855359999, ( 227328-229375):3855396864-3855398911, (229376-231423):3855415296-3855417343, (231424-233471):3855435776-3855437823, (233472-235519):3855648768-3855650815, (235520-237567):3855659008-3855661055, (237568-239615):3855667200-3855669247, (239616-241663):3855675392-3855677439, (241664-243711):3855681536-3855683583, (243712-245759):3855687 680-3855689727, (245760-247807):3855740928-3855742975, (247808-251903):3855722496-3855726591, (251904-253951):3855734784-3855736831, (253952-255999):3855769600-3855771647, (256000-258047):3855745024-3855747071, (258048-260095):3855759360-3855761407, (260096-262143):3855777792-3855779839, (262144-264191):3855761408-3855763455, (26419 2-266239):3855757312-3855759359, (266240-270335):3855800320-3855804415, (270336-274431):3855792128-3855796223, (274432-276479):3855835136-3855837183, (276480-278527):3855865856-3855867903, (278528-280575):3855872000-3855874047, (280576-282623):3855880192-3855882239, (282624-284671):3855892480-3855894527, (284672-286719):3855904768-3 855906815, (286720-288767):3855910912-3855912959, (288768-290815):3855919104-3855921151, (290816-292863):3855951872-3855953919, (292864-294911):3855947776-3855949823, (294912-296959):3855960064-3855962111, (296960-299007):3856050176-3856052223, (299008-301055):3856064512-3856066559, (301056-303103):3856101376-3856103423, (303104-305 151):3856996352-3856998399, (305152-307199):3857031168-3857033215, (307200-309247):3857041408-3857043455, (309248-311295):3857084416-3857086463, (311296-313343):3857057792-3857059839, (313344-315391):3857078272-3857080319, (315392-317439):3857102848-3857104895, (317440-319487):3857123328-3857125375, (319488-321535):3857129472-385713 1519, (321536-323583):3857145856-3857147903, (323584-325631):3857184768-3857186815, (325632-327679):3857258496-3857260543, (327680-331775):3855804416-3855808511, (331776-333823):3855812608-3855814655, (333824-335871):3855828992-3855831039, (335872-337919):3855833088-3855835135, (337920-339967):3855869952-3855871999, (339968-342015): 3855878144-3855880191, (342016-344063):3855902720-3855904767, (344064-348159):3855906816-3855910911, (348160-350207):3855927296-3855929343, (350208-352255):3855953920-3855955967, (352256-354303):3855945728-3855947775, (354304-356351):3855962112-3855964159, (356352-360447):3855984640-3855988735, (360448-362495):3856005120-3856007167, (362496-366591):3856041984-3856046079, (366592-368639):3856056320-3856058367, (368640-370687):3856066560-3856068607, (370688-372735):3856459776-3856461823, (372736-374783):3856998400-3857000447, (374784-376831):3857047552-3857049599, (376832-380927):3857074176-3857078271, (380928-382975):3857096704-3857098751, (382976-385023):38571 41760-3857143807, (385024-387071):3857158144-3857160191, (387072-389119):3857260544-3857262591, (389120-391167):3857270784-3857272831, (391168-393215):3857285120-3857287167, (393216-397311):3855808512-3855812607, (397312-399359):3855796224-3855798271, (399360-401407):3855839232-3855841279, (401408-403455):3855837184-3855839231, (403 456-405503):3855867904-3855869951, (405504-409599):3855874048-3855878143, (409600-411647):3855898624-3855900671, (411648-413695):3855890432-3855892479, (413696-415743):3855937536-3855939583, (415744-417791):3855925248-3855927295, (417792-419839):3855921152-3855923199, (419840-421887):3855941632-3855943679, (421888-423935):3855970304 -3855972351, (423936-425983):3855964160-3855966207, (425984-428031):3856054272-3856056319, (428032-430079):3856058368-3856060415, (430080-432127):3856080896-3856082943, (432128-434175):3856099328-3856101375, (434176-436223):3857004544-3857006591, (436224-438271):3857029120-3857031167, (438272-440319):3857049600-3857051647, (440320-4 42367):3857039360-3857041407, (442368-446463):3857061888-3857065983, (446464-448511):3857080320-3857082367, (448512-450559):3857106944-3857108991, (450560-452607):3857125376-3857127423, (452608-454655):3857139712-3857141759, (454656-456703):3857217536-3857219583, (456704-458751):3857225728-3857227775, (458752-462847):3855886336-3855 890431, (462848-464895):3855931392-3855933439, (464896-466943):3855917056-3855919103, (466944-468991):3855923200-3855925247, (468992-471039):3855966208-3855968255, (471040-473087):3855958016-3855960063, (473088-475135):3855980544-3855982591, (475136-477183):3856068608-3856070655, (477184-479231):3856072704-3856074751, (479232-481279 ):3856093184-3856095231, (481280-483327):3856457728-3856459775, (483328-485375):3856967680-3856969727, (485376-487423):3857014784-3857016831, (487424-489471):3857037312-3857039359, (489472-493567):3857043456-3857047551, (493568-495615):3857055744-3857057791, (495616-497663):3857065984-3857068031, (497664-499711):3857082368-385708441 5, (499712-501759):3857115136-3857117183, (501760-503807):3857133568-3857135615, (503808-505855):3857137664-3857139711, (505856-507903):3857242112-3857244159, (507904-509951):3857272832-3857274879, (509952-511999):3857303552-3857305599, (512000-514047):3857287168-3857289215, (514048-516095):3857336320-3857338367, (516096-518143):385 7573888-3857575935, (518144-520191):3857790976-3857793023, (520192-522239):3857823744-3857825791, (522240-524287):3857944576-3857946623, (524288-526335):3857268736-3857270783, (526336-528383):3857305600-3857307647, (528384-530431):3857319936-3857321983, (530432-532479):3857330176-3857332223, (532480-534527):3857762304-3857764351, (5 34528-536575):3857807360-3857809407, (536576-538623):3857811456-3857813503, (538624-540671):3857936384-3857938431, (540672-542719):3857981440-3857983487, (542720-544767):3857985536-3857987583, (544768-546815):3857991680-3857993727, (546816-550911):3858003968-3858008063, (550912-552959):3858032640-3858034687, (552960-555007):38580654 08-3858067455, (555008-557055):3858079744-3858081791, (557056-559103):3858114560-3858116607, (559104-561151):3858210816-3858212863, (561152-563199):3858214912-3858216959, (563200-565247):3858247680-3858249727, (565248-567295):3858325504-3858327551, (567296-571391):3858335744-3858339839, (571392-573439):3858427904-3858429951, (573440 -575487):3858497536-3858499583, (575488-577535):3858550784-3858552831, (577536-579583):3858593792-3858595839, (579584-581631):3858575360-3858577407, (581632-583679):3858599936-3858601983, (583680-585727):3858622464-3858624511, (585728-587775):3858632704-3858634751, (587776-589823):3858642944-3858644991, (589824-591871):3857326080-38 57328127, (591872-593919):3857367040-3857369087, (593920-595967):3857575936-3857577983, (595968-598015):3857797120-3857799167, (598016-600063):3857809408-3857811455, (600064-602111):3857934336-3857936383, (602112-604159):3857946624-3857948671, (604160-606207):3857975296-3857977343, (606208-608255):3858001920-3858003967, (608256-6103 03):3858022400-3858024447, (610304-612351):3858067456-3858069503, (612352-614399):3858075648-3858077695, (614400-616447):3858087936-3858089983, (616448-618495):3858100224-3858102271, (618496-620543):3858206720-3858208767, (620544-622591):3858204672-3858206719, (622592-624639):3858229248-3858231295, (624640-626687):3858302976-3858305 023, (626688-628735):3858339840-3858341887, (628736-630783):3858370560-3858372607, (630784-634879):3858528256-3858532351, (634880-636927):3858540544-3858542591, (636928-638975):3858573312-3858575359, (638976-641023):3858577408-3858579455, (641024-643071):3858626560-3858628607, (643072-645119):3858630656-3858632703, (645120-647167):3 858657280-3858659327, (647168-651263):3858647040-3858651135, (651264-653311):3858704384-3858706431, (653312-655359):3858794496-3858796543, (655360-657407):3857340416-3857342463, (657408-659455):3857551360-3857553407, (659456-661503):3857752064-3857754111, (661504-663551):3857803264-3857805311, (663552-665599):3857854464-3857856511, (665600-667647):3857950720-3857952767, (667648-669695):3857973248-3857975295, (669696-671743):3857987584-3857989631, (671744-673791):3858008064-3858010111, (673792-675839):3858024448-3858026495, (675840-677887):3858069504-3858071551, (677888-679935):3858083840-3858085887, (679936-681983):3858102272-3858104319, (681984-684031):385820 2624-3858204671, (684032-686079):3858216960-3858219007, (686080-688127):3858286592-3858288639, (688128-690175):3858321408-3858323455, (690176-692223):3858341888-3858343935, (692224-694271):3858358272-3858360319, (694272-696319):3858354176-3858356223, (696320-698367):3858466816-3858468863, (698368-700415):3858544640-3858546687, (7004 16-702463):3858532352-3858534399, (702464-704511):3858581504-3858583551, (704512-706559):3858610176-3858612223, (706560-708607):3858614272-3858616319, (708608-710655):3858638848-3858640895, (710656-714751):3858659328-3858663423, (714752-716799):3858685952-3858687999, (716800-718847):3858681856-3858683903, (718848-720895):3858710528- 3858712575, (ETB2):3858664192, (720896-722943):3857989632-3857991679, (722944-724991):3858020352-3858022399, (724992-727039):3858057216-3858059263, (727040-729087):3858071552-3858073599, (729088-731135):3858096128-3858098175, (731136-733183):3858104320-3858106367, (733184-737279):3858225152-3858229247, (737280-739327):3858305024-385 8307071, (739328-741375):3858352128-3858354175, (741376-743423):3858356224-3858358271, (743424-745471):3858495488-3858497535, (745472-749567):3858536448-3858540543, (749568-751615):3858579456-3858581503, (751616-753663):3858571264-3858573311, (753664-755711):3858606080-3858608127, (755712-757759):3858612224-3858614271, (757760-75980 7):3858640896-3858642943, (759808-761855):3858644992-3858647039, (ETB2):3858629888, (761856-763903):3858667520-3858669567, (763904-765951):3858679808-3858681855, (765952-767999):3858706432-3858708479, (768000-770047):3858651136-3858653183, (770048-772095):3858735104-3858737151, (772096-774143):3858825216-3858827263, (774144-776191): 3858849792-3858851839, (776192-778239):3858857984-3858860031, (778240-782335):3858864128-3858868223, (782336-784383):3858872320-3858874367, (784384-786431):3858907136-3858909183, (786432-788479):3858677760-3858679807, (788480-790527):3858683904-3858685951, (790528-792575):3858702336-3858704383, (792576-794623):3858796544-3858798591, (794624-796671):3858847744-3858849791, (796672-798719):3858851840-3858853887, (798720-800767):3858886656-3858888703, (800768-802815):3858882560-3858884607, (802816-804863):3858870272-3858872319, (804864-806911):3858900992-3858903039, (806912-808959):3858905088-3858907135, (808960-811007):3858972672-3858974719, (811008-813055):38589 89056-3858991103, (813056-815103):3858997248-3858999295, (815104-817151):3859013632-3859015679, (817152-819199):3859036160-3859038207, (819200-821247):3859179520-3859181567, (821248-823295):3859191808-3859193855, (823296-825343):3859212288-3859214335, (825344-827391):3859230720-3859232767, (827392-829439):3859271680-3859273727, (829 440-831487):3859265536-3859267583, (831488-833535):3859283968-3859286015, (833536-835583):3859363840-3859365887, (835584-837631):3859560448-3859562495, (837632-839679):3859599360-3859601407, (839680-841727):3859623936-3859625983, (841728-843775):3859654656-3859656703, (843776-845823):3859695616-3859697663, (845824-847871):3859750912 -3859752959, (847872-851967):3859781632-3859785727, (851968-854015):3858743296-3858745343, (854016-856063):3858833408-3858835455, (856064-858111):3858855936-3858857983, (858112-860159):3858862080-3858864127, (860160-862207):3858876416-3858878463, (862208-864255):3858868224-3858870271, (864256-866303):3858894848-3858896895, (866304-8 68351):3858903040-3858905087, (868352-870399):3858941952-3858943999, (870400-872447):3858991104-3858993151, (872448-874495):3858999296-3859001343, (874496-884735):3859003392-3859013631, (884736-886783):3859019776-3859021823, (886784-888831):3859204096-3859206143, (888832-890879):3859208192-3859210239, (890880-892927):3859238912-3859 240959, (892928-894975):3859261440-3859263487, (894976-897023):3859306496-3859308543, (897024-899071):3859286016-3859288063, (899072-901119):3859353600-3859355647, (901120-903167):3859582976-3859585023, (903168-905215):3859632128-3859634175, (905216-909311):3859666944-3859671039, (909312-911359):3859660800-3859662847, (911360-913407 ):3859685376-3859687423, (913408-915455):3859681280-3859683327, (915456-917503):3859742720-3859744767, (917504-919551):3858874368-3858876415, (919552-921599):3858884608-3858886655, (921600-923647):3858892800-3858894847, (923648-925695):3858933760-3858935807, (925696-927743):3858993152-3858995199, (927744-929791):3859001344-385900339 1, (929792-931839):3859015680-3859017727, (931840-935935):3859023872-3859027967, (935936-937983):3859038208-3859040255, (937984-940031):3859189760-3859191807, (940032-942079):3859210240-3859212287, (942080-944127):3859275776-3859277823, (944128-946175):3859273728-3859275775, (946176-948223):3859288064-3859290111, (948224-950271):385 9402752-3859404799, (950272-952319):3859570688-3859572735, (952320-954367):3859597312-3859599359, (954368-956415):3859601408-3859603455, (956416-958463):3859636224-3859638271, (958464-960511):3859662848-3859664895, (960512-962559):3859683328-3859685375, (962560-964607):3859699712-3859701759, (964608-966655):3859748864-3859750911, (9 66656-970751):3859789824-3859793919, (970752-972799):3859798016-3859800063, (972800-974847):3859828736-3859830783, (974848-978943):3859808256-3859812351, (978944-980991):3859855360-3859857407, (980992-983039):3859884032-3859886079, (983040-985087):3858978816-3858980863, (985088-987135):3859017728-3859019775, (987136-989183):38590402 56-3859042303, (989184-991231):3859206144-3859208191, (991232-995327):3859257344-3859261439, (995328-997375):3859263488-3859265535, (997376-999423):3859290112-3859292159, (999424-1001471):3859453952-3859455999, (1001472-1003519):3859591168-3859593215, (1003520-1005567):3859585024-3859587071, (1005568-1007615):3859634176-3859636223, debugfs: ncheck 946398 Inode Pathname 946398 /O/0/d15/168840719 </pre> </div></div> LU-11638

lfs fid2path should list open-unlinked files

Improvement Major Reopened Unresolved Andreas Dilger Mahmoud Hanafi medium Wed, 7 Nov 2018 17:38:16 +0000 Wed, 24 Nov 2021 23:50:40 +0000 Lustre 2.10.5 0 7 <p>Is this one of the filesystems related to <a href="https://jira.whamcloud.com/browse/LU-11548" title="increase default readahead tunables to match common settings" class="issue-link" data-issue-key="LU-11548"><del>LU-11548</del></a>? One of the problems seen in that case was multiple hard links to the same OST object. If that is the case here, it is possible that the "real" object was removed, but a hard linked name was left behind and the space was not freed. </p> <p>Another possibility is that the object was migrated to a new MDT and the parent FID on the object was not updated. You might be able to search for large files in the directory (possibly belonging to UID 12137 and/or GID 21205) to see if that is the case?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs find /nobackupp8 -size +3T -print0 | xargs -0 ls -l </pre> </div></div> <p>I assume there were not any other events around this time (eg. MDS or OSS crash)? That shouldn't cause an unlinked file to leave orphan objects behind, but knowing the circumstances around this file may help. </p> <p>The orphan object(s) can be linked to <tt>/nobackupp8/.lustre/lost+found/</tt> and/or unlinked by running <tt>lctl lfsck_start -A -t layout</tt> on the MDS (assuming this is not having other issues at the current time). Alternately, instead of running LFSCK, if the <tt>lfs find</tt> does not turn up any candidate files, you could mount the OST as type ldiskfs and delete the objects manually. </p> <p>This filesystem was not related to <a href="https://jira.whamcloud.com/browse/LU-11548" title="increase default readahead tunables to match common settings" class="issue-link" data-issue-key="LU-11548"><del>LU-11548</del></a>. The only that may have happened here is the file was delete while it was still being written to from a different client. And then that write thread was killed.</p> <p>Is it safe to mount the ost as ldiskfs and delete the object while the OST is mounted as lustre?</p> <p>Officially, no. In theory, yes it is possible to do that (I sometimes do concurrent ldiskfs mounts on my home filesystem for debugging purposes), but it is not something we have officially tested. If you have a quiet time and could "expect" a crash, but are happy if it works without problems or interference with jobs, then I'd say it should <em>probably</em> be OK to do that. Otherwise, I'd suggest to wait until there is a scheduled outage window and try it then. At worst it would crash, but that is not much worse than just unmounting the OST (causing IO to block), remount as ldiskfs and remove the object, then remount the OST and go through recovery.</p> <p>I was able to delete the object after umounting OST and remount via ldiskfs.</p> <p>Please close this issue.</p> <p>Good news - thanks!</p> <p>Reopening this issue because we don't know what the root cause is. We <em>should</em> be able to handle cleanup of open-unlinked files properly, and/or files that are unlinked from the MDT but held open when the MDS crashed.</p> <p>I did a small test and saw that <tt>lfs fid2path</tt> does not report anything for an open-unlinked file:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>client$ lfs path2fid /myth/tmp/1M [0x200003ab4:0x435:0x0] client$ sleep 10000 &lt; /myth/tmp/1M &amp; [1] 13806 client$ sudo lfs fid2path /myth [0x200003ab4:0x435:0x0] /myth/tmp/1M mds# lctl get_param mdt.*.exports.*.open_files mdt.myth-MDT0000.exports.192.168.20.159@tcp.open_files=[0x200003ab4:0x435:0x0] client$ rm /myth/tmp/1M mds# lctl get_param mdt.*.exports.*.open_files mdt.myth-MDT0000.exports.192.168.20.159@tcp.open_files=[0x200003ab4:0x435:0x0] client$ sudo lfs fid2path /myth [0x200003ab4:0x435:0x0] lfs fid2path: cannot find '[0x200003ab4:0x435:0x0]': No such file or directory client$ lsof /myth COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sleep 13806 mythtv 0r REG 35,632494 1048576 144115440203858997 /myth/tmp/1M (deleted) </pre> </div></div> <p>If this happens again in the future, it would be useful to check the <tt>mdt.&#42;.exports.&#42;.open_files</tt> list to see if the FID is listed there, which also lists the client NID that is holding it open, and you could use <tt>lsof</tt> on the client to find and kill the offending process (note that <tt>lsof</tt> will only report file handles open on the local node).</p> <p>It would be useful if <tt>lfs fid2path</tt> listed these open-unlinked files directly, including the client NID and appends <tt>(deleted)</tt> as lsof does.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33629" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33629</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11638" title="lfs fid2path should list open-unlinked files" class="issue-link" data-issue-key="LU-11638">LU-11638</a> admin: debug space management/missing objects<br/> Project: doc/manual<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 46989d10fe59fa2df6d38885cf53839da9add6d4</p> <p>Joseph Gmitter (jgmitter@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33629/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33629/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11638" title="lfs fid2path should list open-unlinked files" class="issue-link" data-issue-key="LU-11638">LU-11638</a> admin: debug space management/missing objects<br/> Project: doc/manual<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5e609ea889c9626dfe558e170d78ab74dda42230</p> Duplicate LU-11687 Related LU-13527 Development Rank 1|i005yf: Rank (Obsolete) 9223372036854775807 Severity