[LU-11638] lfs fid2path should list open-unlinked files

• Type: Improvement
• Resolution: Unresolved
• Priority: Major
• Affects Version/s: Lustre 2.10.5
• Labels: debug medium usability
• Severity: 3
• Assignee: Andreas Dilger
• Reporter: Mahmoud Hanafi

Description

We delete a large file 15TB (4 stripes), but the objects are not deleting.

pfe24 ~ # lfs fid2path /nobackupp8 0x360635236:0xbd:0x0
fid2path: error on FID 0x360635236:0xbd:0x0: No such file or directory

debugfs:  stat <946398>
Inode: 946398   Type: regular    Mode:  0666   Flags: 0x80000
Generation: 1914950395    Version: 0x00000030:005b5f49
User: 12137   Group: 21205   Project:     0   Size: 3857685938176
File ACL: 0
Links: 1   Blockcount: 7534584632
Fragment:  Address: 0    Number: 0    Size: 0
 ctime: 0x5be3096c:00000000 -- Wed Nov  7 07:49:00 2018
 atime: 0x5be1ae0c:00000000 -- Tue Nov  6 07:06:52 2018
 mtime: 0x5be3096c:00000000 -- Wed Nov  7 07:49:00 2018
crtime: 0x5be1ace3:1d2d1844 -- Tue Nov  6 07:01:55 2018
Size of extra inode fields: 32
Extended attributes:
  trusted.lma (64)
  lma: fid=[0x100000000:0xa104e0f:0x0] compat=18 incompat=0
  fid: parent=[0x360635236:0xbd:0x0] stripe=3 stripe_size=1048576 stripe_count=4
EXTENTS:
(ETB0):234918342, (ETB1):234918343, (ETB2):234918298, (0-2047):3853318144-3853320191, (2048-4095):3854127104-3854129151, (4096-6143):3854174208-3854176255, (6144-8191):3854239744-3854241791, (8192-10239):3854297088-3854299135, (10240-12287):3854319616-3854321663, (12288-14335):3854405632-3854407679, (14336-16383):3854417920-38544199
67, (16384-18431):3854473216-3854475263, (18432-20479):3854772224-3854774271, (20480-22527):3854839808-3854841855, (22528-24575):3854944256-3854946303, (24576-26623):3855372288-3855374335, (26624-28671):3855370240-3855372287, (28672-30719):3855405056-3855407103, (30720-34815):3855431680-3855435775, (34816-36863):3855423488-385542553
5, (36864-38911):3855654912-3855656959, (38912-40959):3855663104-3855665151, (40960-43007):3855669248-3855671295, (43008-45055):3855673344-3855675391, (45056-47103):3855691776-3855693823, (47104-49151):3855702016-3855704063, (49152-53247):3855718400-3855722495, (53248-55295):3855730688-3855732735, (55296-57343):3855753216-3855755263
, (57344-59391):3855747072-3855749119, (59392-61439):3855765504-3855767551, (61440-65535):3855783936-3855788031, (65536-67583):3854090240-3854092287, (67584-69631):3854106624-3854108671, (69632-71679):3854204928-3854206975, (71680-73727):3854295040-3854297087, (73728-75775):3854374912-3854376959, (75776-77823):3854413824-3854415871,
 (77824-79871):3854419968-3854422015, (79872-81919):3854712832-3854714879, (81920-83967):3854811136-3854813183, (83968-86015):3854845952-3854847999, (86016-88063):3854950400-3854952447, (88064-90111):3855187968-3855190015, (90112-92159):3855368192-3855370239, (92160-94207):3855403008-3855405055, (94208-96255):3855409152-3855411199, 
(96256-98303):3855417344-3855419391, (98304-100351):3855448064-3855450111, (100352-102399):3855650816-3855652863, (102400-104447):3855665152-3855667199, (104448-106495):3855683584-3855685631, (106496-108543):3855689728-3855691775, (108544-110591):3855708160-3855710207, (110592-112639):3855714304-3855716351, (112640-114687):385572659
2-3855728639, (114688-116735):3855749120-3855751167, (116736-118783):3855773696-3855775743, (118784-120831):3855767552-3855769599, (120832-124927):3855788032-3855792127, (124928-126975):3855818752-3855820799, (126976-129023):3855831040-3855833087, (129024-131071):3855855616-3855857663, (131072-133119):3853316096-3853318143, (133120-
135167):3854098432-3854100479, (135168-137215):3854112768-3854114815, (137216-139263):3854178304-3854180351, (139264-141311):3854266368-3854268415, (141312-143359):3854292992-3854295039, (143360-145407):3854315520-3854317567, (145408-147455):3854428160-3854430207, (147456-149503):3854409728-3854411775, (149504-151551):3854452736-385
4454783, (151552-153599):3854471168-3854473215, (153600-155647):3854755840-3854757887, (155648-157695):3854835712-3854837759, (157696-159743):3854942208-3854944255, (159744-161791):3855101952-3855103999, (161792-163839):3855355904-3855357951, (163840-165887):3855413248-3855415295, (165888-167935):3855441920-3855443967, (167936-16998
3):3855421440-3855423487, (169984-172031):3855642624-3855644671, (172032-174079):3855661056-3855663103, (174080-176127):3855671296-3855673343, (176128-180223):3855677440-3855681535, (180224-182271):3855695872-3855697919, (182272-184319):3855699968-3855702015, (184320-188415):3855704064-3855708159, (188416-190463):3855710208-38557122
55, (190464-194559):3855736832-3855740927, (194560-196607):3855728640-3855730687, (196608-198655):3854104576-3854106623, (198656-200703):3854143488-3854145535, (200704-202751):3854192640-3854194687, (202752-204799):3854264320-3854266367, (204800-206847):3854299136-3854301183, (206848-208895):3854385152-3854387199, (208896-210943):38
54407680-3854409727, (210944-212991):3854411776-3854413823, (212992-215039):3854454784-3854456831, (215040-217087):3854747648-3854749695, (217088-219135):3854813184-3854815231, (219136-221183):3854850048-3854852095, (221184-223231):3854952448-3854954495, (223232-225279):3855351808-3855353855, (225280-227327):3855357952-3855359999, (
227328-229375):3855396864-3855398911, (229376-231423):3855415296-3855417343, (231424-233471):3855435776-3855437823, (233472-235519):3855648768-3855650815, (235520-237567):3855659008-3855661055, (237568-239615):3855667200-3855669247, (239616-241663):3855675392-3855677439, (241664-243711):3855681536-3855683583, (243712-245759):3855687
680-3855689727, (245760-247807):3855740928-3855742975, (247808-251903):3855722496-3855726591, (251904-253951):3855734784-3855736831, (253952-255999):3855769600-3855771647, (256000-258047):3855745024-3855747071, (258048-260095):3855759360-3855761407, (260096-262143):3855777792-3855779839, (262144-264191):3855761408-3855763455, (26419
2-266239):3855757312-3855759359, (266240-270335):3855800320-3855804415, (270336-274431):3855792128-3855796223, (274432-276479):3855835136-3855837183, (276480-278527):3855865856-3855867903, (278528-280575):3855872000-3855874047, (280576-282623):3855880192-3855882239, (282624-284671):3855892480-3855894527, (284672-286719):3855904768-3
855906815, (286720-288767):3855910912-3855912959, (288768-290815):3855919104-3855921151, (290816-292863):3855951872-3855953919, (292864-294911):3855947776-3855949823, (294912-296959):3855960064-3855962111, (296960-299007):3856050176-3856052223, (299008-301055):3856064512-3856066559, (301056-303103):3856101376-3856103423, (303104-305
151):3856996352-3856998399, (305152-307199):3857031168-3857033215, (307200-309247):3857041408-3857043455, (309248-311295):3857084416-3857086463, (311296-313343):3857057792-3857059839, (313344-315391):3857078272-3857080319, (315392-317439):3857102848-3857104895, (317440-319487):3857123328-3857125375, (319488-321535):3857129472-385713
1519, (321536-323583):3857145856-3857147903, (323584-325631):3857184768-3857186815, (325632-327679):3857258496-3857260543, (327680-331775):3855804416-3855808511, (331776-333823):3855812608-3855814655, (333824-335871):3855828992-3855831039, (335872-337919):3855833088-3855835135, (337920-339967):3855869952-3855871999, (339968-342015):
3855878144-3855880191, (342016-344063):3855902720-3855904767, (344064-348159):3855906816-3855910911, (348160-350207):3855927296-3855929343, (350208-352255):3855953920-3855955967, (352256-354303):3855945728-3855947775, (354304-356351):3855962112-3855964159, (356352-360447):3855984640-3855988735, (360448-362495):3856005120-3856007167,
 (362496-366591):3856041984-3856046079, (366592-368639):3856056320-3856058367, (368640-370687):3856066560-3856068607, (370688-372735):3856459776-3856461823, (372736-374783):3856998400-3857000447, (374784-376831):3857047552-3857049599, (376832-380927):3857074176-3857078271, (380928-382975):3857096704-3857098751, (382976-385023):38571
41760-3857143807, (385024-387071):3857158144-3857160191, (387072-389119):3857260544-3857262591, (389120-391167):3857270784-3857272831, (391168-393215):3857285120-3857287167, (393216-397311):3855808512-3855812607, (397312-399359):3855796224-3855798271, (399360-401407):3855839232-3855841279, (401408-403455):3855837184-3855839231, (403
456-405503):3855867904-3855869951, (405504-409599):3855874048-3855878143, (409600-411647):3855898624-3855900671, (411648-413695):3855890432-3855892479, (413696-415743):3855937536-3855939583, (415744-417791):3855925248-3855927295, (417792-419839):3855921152-3855923199, (419840-421887):3855941632-3855943679, (421888-423935):3855970304
-3855972351, (423936-425983):3855964160-3855966207, (425984-428031):3856054272-3856056319, (428032-430079):3856058368-3856060415, (430080-432127):3856080896-3856082943, (432128-434175):3856099328-3856101375, (434176-436223):3857004544-3857006591, (436224-438271):3857029120-3857031167, (438272-440319):3857049600-3857051647, (440320-4
42367):3857039360-3857041407, (442368-446463):3857061888-3857065983, (446464-448511):3857080320-3857082367, (448512-450559):3857106944-3857108991, (450560-452607):3857125376-3857127423, (452608-454655):3857139712-3857141759, (454656-456703):3857217536-3857219583, (456704-458751):3857225728-3857227775, (458752-462847):3855886336-3855
890431, (462848-464895):3855931392-3855933439, (464896-466943):3855917056-3855919103, (466944-468991):3855923200-3855925247, (468992-471039):3855966208-3855968255, (471040-473087):3855958016-3855960063, (473088-475135):3855980544-3855982591, (475136-477183):3856068608-3856070655, (477184-479231):3856072704-3856074751, (479232-481279
):3856093184-3856095231, (481280-483327):3856457728-3856459775, (483328-485375):3856967680-3856969727, (485376-487423):3857014784-3857016831, (487424-489471):3857037312-3857039359, (489472-493567):3857043456-3857047551, (493568-495615):3857055744-3857057791, (495616-497663):3857065984-3857068031, (497664-499711):3857082368-385708441
5, (499712-501759):3857115136-3857117183, (501760-503807):3857133568-3857135615, (503808-505855):3857137664-3857139711, (505856-507903):3857242112-3857244159, (507904-509951):3857272832-3857274879, (509952-511999):3857303552-3857305599, (512000-514047):3857287168-3857289215, (514048-516095):3857336320-3857338367, (516096-518143):385
7573888-3857575935, (518144-520191):3857790976-3857793023, (520192-522239):3857823744-3857825791, (522240-524287):3857944576-3857946623, (524288-526335):3857268736-3857270783, (526336-528383):3857305600-3857307647, (528384-530431):3857319936-3857321983, (530432-532479):3857330176-3857332223, (532480-534527):3857762304-3857764351, (5
34528-536575):3857807360-3857809407, (536576-538623):3857811456-3857813503, (538624-540671):3857936384-3857938431, (540672-542719):3857981440-3857983487, (542720-544767):3857985536-3857987583, (544768-546815):3857991680-3857993727, (546816-550911):3858003968-3858008063, (550912-552959):3858032640-3858034687, (552960-555007):38580654
08-3858067455, (555008-557055):3858079744-3858081791, (557056-559103):3858114560-3858116607, (559104-561151):3858210816-3858212863, (561152-563199):3858214912-3858216959, (563200-565247):3858247680-3858249727, (565248-567295):3858325504-3858327551, (567296-571391):3858335744-3858339839, (571392-573439):3858427904-3858429951, (573440
-575487):3858497536-3858499583, (575488-577535):3858550784-3858552831, (577536-579583):3858593792-3858595839, (579584-581631):3858575360-3858577407, (581632-583679):3858599936-3858601983, (583680-585727):3858622464-3858624511, (585728-587775):3858632704-3858634751, (587776-589823):3858642944-3858644991, (589824-591871):3857326080-38
57328127, (591872-593919):3857367040-3857369087, (593920-595967):3857575936-3857577983, (595968-598015):3857797120-3857799167, (598016-600063):3857809408-3857811455, (600064-602111):3857934336-3857936383, (602112-604159):3857946624-3857948671, (604160-606207):3857975296-3857977343, (606208-608255):3858001920-3858003967, (608256-6103
03):3858022400-3858024447, (610304-612351):3858067456-3858069503, (612352-614399):3858075648-3858077695, (614400-616447):3858087936-3858089983, (616448-618495):3858100224-3858102271, (618496-620543):3858206720-3858208767, (620544-622591):3858204672-3858206719, (622592-624639):3858229248-3858231295, (624640-626687):3858302976-3858305
023, (626688-628735):3858339840-3858341887, (628736-630783):3858370560-3858372607, (630784-634879):3858528256-3858532351, (634880-636927):3858540544-3858542591, (636928-638975):3858573312-3858575359, (638976-641023):3858577408-3858579455, (641024-643071):3858626560-3858628607, (643072-645119):3858630656-3858632703, (645120-647167):3
858657280-3858659327, (647168-651263):3858647040-3858651135, (651264-653311):3858704384-3858706431, (653312-655359):3858794496-3858796543, (655360-657407):3857340416-3857342463, (657408-659455):3857551360-3857553407, (659456-661503):3857752064-3857754111, (661504-663551):3857803264-3857805311, (663552-665599):3857854464-3857856511, 
(665600-667647):3857950720-3857952767, (667648-669695):3857973248-3857975295, (669696-671743):3857987584-3857989631, (671744-673791):3858008064-3858010111, (673792-675839):3858024448-3858026495, (675840-677887):3858069504-3858071551, (677888-679935):3858083840-3858085887, (679936-681983):3858102272-3858104319, (681984-684031):385820
2624-3858204671, (684032-686079):3858216960-3858219007, (686080-688127):3858286592-3858288639, (688128-690175):3858321408-3858323455, (690176-692223):3858341888-3858343935, (692224-694271):3858358272-3858360319, (694272-696319):3858354176-3858356223, (696320-698367):3858466816-3858468863, (698368-700415):3858544640-3858546687, (7004
16-702463):3858532352-3858534399, (702464-704511):3858581504-3858583551, (704512-706559):3858610176-3858612223, (706560-708607):3858614272-3858616319, (708608-710655):3858638848-3858640895, (710656-714751):3858659328-3858663423, (714752-716799):3858685952-3858687999, (716800-718847):3858681856-3858683903, (718848-720895):3858710528-
3858712575, (ETB2):3858664192, (720896-722943):3857989632-3857991679, (722944-724991):3858020352-3858022399, (724992-727039):3858057216-3858059263, (727040-729087):3858071552-3858073599, (729088-731135):3858096128-3858098175, (731136-733183):3858104320-3858106367, (733184-737279):3858225152-3858229247, (737280-739327):3858305024-385
8307071, (739328-741375):3858352128-3858354175, (741376-743423):3858356224-3858358271, (743424-745471):3858495488-3858497535, (745472-749567):3858536448-3858540543, (749568-751615):3858579456-3858581503, (751616-753663):3858571264-3858573311, (753664-755711):3858606080-3858608127, (755712-757759):3858612224-3858614271, (757760-75980
7):3858640896-3858642943, (759808-761855):3858644992-3858647039, (ETB2):3858629888, (761856-763903):3858667520-3858669567, (763904-765951):3858679808-3858681855, (765952-767999):3858706432-3858708479, (768000-770047):3858651136-3858653183, (770048-772095):3858735104-3858737151, (772096-774143):3858825216-3858827263, (774144-776191):
3858849792-3858851839, (776192-778239):3858857984-3858860031, (778240-782335):3858864128-3858868223, (782336-784383):3858872320-3858874367, (784384-786431):3858907136-3858909183, (786432-788479):3858677760-3858679807, (788480-790527):3858683904-3858685951, (790528-792575):3858702336-3858704383, (792576-794623):3858796544-3858798591,
 (794624-796671):3858847744-3858849791, (796672-798719):3858851840-3858853887, (798720-800767):3858886656-3858888703, (800768-802815):3858882560-3858884607, (802816-804863):3858870272-3858872319, (804864-806911):3858900992-3858903039, (806912-808959):3858905088-3858907135, (808960-811007):3858972672-3858974719, (811008-813055):38589
89056-3858991103, (813056-815103):3858997248-3858999295, (815104-817151):3859013632-3859015679, (817152-819199):3859036160-3859038207, (819200-821247):3859179520-3859181567, (821248-823295):3859191808-3859193855, (823296-825343):3859212288-3859214335, (825344-827391):3859230720-3859232767, (827392-829439):3859271680-3859273727, (829
440-831487):3859265536-3859267583, (831488-833535):3859283968-3859286015, (833536-835583):3859363840-3859365887, (835584-837631):3859560448-3859562495, (837632-839679):3859599360-3859601407, (839680-841727):3859623936-3859625983, (841728-843775):3859654656-3859656703, (843776-845823):3859695616-3859697663, (845824-847871):3859750912
-3859752959, (847872-851967):3859781632-3859785727, (851968-854015):3858743296-3858745343, (854016-856063):3858833408-3858835455, (856064-858111):3858855936-3858857983, (858112-860159):3858862080-3858864127, (860160-862207):3858876416-3858878463, (862208-864255):3858868224-3858870271, (864256-866303):3858894848-3858896895, (866304-8
68351):3858903040-3858905087, (868352-870399):3858941952-3858943999, (870400-872447):3858991104-3858993151, (872448-874495):3858999296-3859001343, (874496-884735):3859003392-3859013631, (884736-886783):3859019776-3859021823, (886784-888831):3859204096-3859206143, (888832-890879):3859208192-3859210239, (890880-892927):3859238912-3859
240959, (892928-894975):3859261440-3859263487, (894976-897023):3859306496-3859308543, (897024-899071):3859286016-3859288063, (899072-901119):3859353600-3859355647, (901120-903167):3859582976-3859585023, (903168-905215):3859632128-3859634175, (905216-909311):3859666944-3859671039, (909312-911359):3859660800-3859662847, (911360-913407
):3859685376-3859687423, (913408-915455):3859681280-3859683327, (915456-917503):3859742720-3859744767, (917504-919551):3858874368-3858876415, (919552-921599):3858884608-3858886655, (921600-923647):3858892800-3858894847, (923648-925695):3858933760-3858935807, (925696-927743):3858993152-3858995199, (927744-929791):3859001344-385900339
1, (929792-931839):3859015680-3859017727, (931840-935935):3859023872-3859027967, (935936-937983):3859038208-3859040255, (937984-940031):3859189760-3859191807, (940032-942079):3859210240-3859212287, (942080-944127):3859275776-3859277823, (944128-946175):3859273728-3859275775, (946176-948223):3859288064-3859290111, (948224-950271):385
9402752-3859404799, (950272-952319):3859570688-3859572735, (952320-954367):3859597312-3859599359, (954368-956415):3859601408-3859603455, (956416-958463):3859636224-3859638271, (958464-960511):3859662848-3859664895, (960512-962559):3859683328-3859685375, (962560-964607):3859699712-3859701759, (964608-966655):3859748864-3859750911, (9
66656-970751):3859789824-3859793919, (970752-972799):3859798016-3859800063, (972800-974847):3859828736-3859830783, (974848-978943):3859808256-3859812351, (978944-980991):3859855360-3859857407, (980992-983039):3859884032-3859886079, (983040-985087):3858978816-3858980863, (985088-987135):3859017728-3859019775, (987136-989183):38590402
56-3859042303, (989184-991231):3859206144-3859208191, (991232-995327):3859257344-3859261439, (995328-997375):3859263488-3859265535, (997376-999423):3859290112-3859292159, (999424-1001471):3859453952-3859455999, (1001472-1003519):3859591168-3859593215, (1003520-1005567):3859585024-3859587071, (1005568-1007615):3859634176-3859636223, 
debugfs:  ncheck 946398
Inode   Pathname
946398  /O/0/d15/168840719

 

Issue Links

• is duplicated by: LU-11687 allow "lfs fid2path" on open-unlinked files
• is related to: LU-13527 allow OST FID lookup via lfs fid2path

Comments

[Gerrit Updater]

Joseph Gmitter ([e-mail]) merged in patch https://review.whamcloud.com/33629/
Subject: LU-11638 admin: debug space management/missing objects
Project: doc/manual
Branch: master
Current Patch Set:
Commit: 5e609ea889c9626dfe558e170d78ab74dda42230

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33629
Subject: LU-11638 admin: debug space management/missing objects
Project: doc/manual
Branch: master
Current Patch Set: 1
Commit: 46989d10fe59fa2df6d38885cf53839da9add6d4

[Andreas Dilger]

Reopening this issue because we don't know what the root cause is. We should be able to handle cleanup of open-unlinked files properly, and/or files that are unlinked from the MDT but held open when the MDS crashed.

I did a small test and saw that lfs fid2path does not report anything for an open-unlinked file:

client$ lfs path2fid /myth/tmp/1M
[0x200003ab4:0x435:0x0]
client$ sleep 10000 < /myth/tmp/1M &
[1] 13806
client$ sudo lfs fid2path /myth [0x200003ab4:0x435:0x0]
/myth/tmp/1M
mds# lctl get_param mdt.*.exports.*.open_files
[e-mail]_files=[0x200003ab4:0x435:0x0]
client$ rm /myth/tmp/1M
mds# lctl get_param mdt.*.exports.*.open_files
[e-mail]_files=[0x200003ab4:0x435:0x0]
client$ sudo lfs fid2path /myth [0x200003ab4:0x435:0x0]
lfs fid2path: cannot find '[0x200003ab4:0x435:0x0]': No such file or directory
client$ lsof /myth
COMMAND   PID   USER   FD   TYPE    DEVICE SIZE/OFF               NODE NAME
sleep   13806 mythtv    0r   REG 35,632494  1048576 144115440203858997 /myth/tmp/1M (deleted)

If this happens again in the future, it would be useful to check the mdt.*.exports.*.open_files list to see if the FID is listed there, which also lists the client NID that is holding it open, and you could use lsof on the client to find and kill the offending process (note that lsof will only report file handles open on the local node).

It would be useful if lfs fid2path listed these open-unlinked files directly, including the client NID and appends (deleted) as lsof does.