[LU-11625] ls -l returns "ls: cannot access Invalid argument" Created: 06/Nov/18 Updated: 19/Mar/19 Resolved: 23/Jan/19 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.10.5 |
| Fix Version/s: | Lustre 2.13.0, Lustre 2.10.7, Lustre 2.12.1 |
| Type: | Bug | Priority: | Major |
| Reporter: | Mahmoud Hanafi | Assignee: | Alex Zhuravlev |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Attachments: |
Eta.0000707880.data.obj_stats
Eta.0000707880.data.stripeinfo
ls_client_debug.out
s219.dk.out.gz
|
||||||||
| Issue Links: |
|
||||||||
| Severity: | 2 | ||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||
| Description |
|
After upgrading from 2.7 to 2.10.5 unable to access 97050 files all in the same directory
bash-4.3$ lfs getstripe -d .
stripe_count: -1 stripe_size: 1048576 stripe_offset: -1
bash-4.3$ lfs path2fid .
[0x224179d4c:0x1a0a:0x0]
bash-4.3$ ls -l Eta.0000707880.data
ls: cannot access 'Eta.0000707880.data': Invalid argument
bash-4.3$ lfs path2fid Eta.0000707880.data
[0x224179d4c:0x2883:0x0]
Attaching lfs getstripe out and stat of each object. |
| Comments |
| Comment by Mahmoud Hanafi [ 06/Nov/18 ] |
|
adding client side debug.ls_client_debug.out stat of of file on the mdt
debugfs: stat Eta.0000707880.data
Inode: 349146206 Type: regular Mode: 0555 Flags: 0x0
Generation: 71334841 Version: 0x00000136:cc57dddd
User: 9440 Group: 26209 Size: 0
File ACL: 174575058
Links: 1 Blockcount: 8
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x5bb7c237:00000000 -- Fri Oct 5 12:57:43 2018
atime: 0x59257cae:00000000 -- Wed May 24 05:29:34 2017
mtime: 0x527b0182:00000000 -- Wed Nov 6 18:57:06 2013
crtime: 0x527b0182:df41fdb0 -- Wed Nov 6 18:57:06 2013
Size of extra inode fields: 28
Extended attributes:
trusted.lma (64)
lma: fid=[0x224179d4c:0x2883:0x0] compat=0 incompat=0
fid: parent=[0:0x0:0x0] stripe=0 stripe_size=0 stripe_count=0
trusted.link (61)
trusted.lov (3872)
BLOCKS:
Stat of the parent dir
debugfs: stat .
Inode: 349091160 Type: directory Mode: 0555 Flags: 0x1000
Generation: 71144175 Version: 0x00000136:cc54e27e
User: 9440 Group: 26209 Project: 0 Size: 6987776
File ACL: 0
Links: 3 Blockcount: 13672
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x5bd9f559:00000000 -- Wed Oct 31 11:32:57 2018
atime: 0x5be13846:00000000 -- Mon Nov 5 22:44:22 2018
mtime: 0x5bd9f559:00000000 -- Wed Oct 31 11:32:57 2018
crtime: 0x527ae5d4:43177c18 -- Wed Nov 6 16:59:00 2013
Size of extra inode fields: 32
Extended attributes:
trusted.lma (64)
lma: fid=[0x224179d4c:0x1a0a:0x0] compat=0 incompat=0
fid: parent=[0:0x0:0x0] stripe=0 stripe_size=0 stripe_count=0
trusted.link (56)
trusted.lov (32) = d0 0b d1 0b 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ff ff ff ff
BLOCKS:
(0):174581591, (1-2):174581651-174581652, (3):174581720, (4):174581765, (5):174581834, (6):174581903, (7):174581944, (8):174582005, (9):174582012, (10):174582181, (11):174582208, (IND):174582295, (12):174582296, (13):174582310, (14):174582354, (15):174582398, (16):174582466, (17)
:174582472, (18):174572378, (19):174572471, (20):174572582, (21):174572632, (22):174572640, (23):174572682, (24):174572719, (25):174572873, (26):174572886, (27):174572942, (28):174572947, (29):174572972, (30):174573039, (31):174573068, (32):174573188, (33):174573190, (34):1745732
01, (35):174573339, (36):174573417, (37):174573480, (38):174573501, (39):174573692, (40):174573740, (41):174573764, (42):174573829, (43):174573888, (44):174574000, (45):174574043, (46):174574098, (47):174574161, (48):174574171, (49):174574202, (50):174574251, (51):174574254, (52)
:174574270, (53):174574305, (54):174574348, (55):174574443, (56):174574480, (57):174574680, (58):174574689, (59):174574739, (60):174574825, (61):174574844, (62):174574890, (63):174574904, (64):174574945, (65):174575104, (66):174575148, (67):174575430, (68):174575474, (69):1745755
38, (70):174575554, (71):174575625, (72):174575706, (73):174575725, (74):174575876, (75):174575890, (76):174575930, (77):174576079, (78):174576111, (79):174576140, (80):174576164, (81):174576238, (82):174576252, (83):174576267, (84):174576313, (85):174576356, (86):174576459, (87)
:174576492, (88):174576577, (89):174576607, (90):174576638, (91):174576702, (92):174576704, (93):174576747, (94):174576835, (95):174576869, (96):174576893, (97):174576961, (98):174576988, (99):174577048, (100):174577058, (101):174577106, (102):174577178, (103):174577208, (104):17
4577421, (105):174577454, (106):174577476, (107):174577541, (108):174577571, (109):174577597, (110):174577661, (111):174577680, (112):174577717, (113):174577732, (114):174577742, (115):174577884, (116):174577935, (117):174577951, (118):174578026, (119):174578039, (120):174578181,
(121):174578228, (122):174578257, (123):174578425, (124):174578602, (125):174578668, (126):174578673, (127):174578755, (128):174578925, (129):174578950, (130):174578998, (131):174579092, (132):174579199, (133):174579215, (134):174579217, (135):174579281, (136):174579347, (137):1
74579446, (138):174579659, (139):174579682, (140):174579706, (141):174579799, (142):174579802, (143):174579874, (144):174579941, (145):174580034, (146):174580104, (147):174580242, (148):174580244, (149):174580295, (150):174580337, (151):174600817, (152):174600709, (153):174600730
, (154):174600787, (155):174600501, (156):174600590, (157):174599903, (158):174599907, (159):174599977, (160):174600006, (161):174600024, (162):174600035, (163):174600086, (164):174600100, (165):174600121, (166):174600125, (167):174600180, (168):174600252, (169):174600321, (170):
174600376, (171):174600399, (172):174600407, (173):174598914, (174):174598964, (175):174598976, (176):174599088, (177):174599162, (178):174599202, (179):174599206, (180):174599260, (181):174599265, (182):174599283, (183):174599293, (184):174599353, (185):174599362, (186):17459955
2, (187):174599581, (188):174599639, (189):174599650, (190):174599745, (191):174599856, (192):174596824, (193):174596836, (194):174597008, (195):174597027, (196):174597040, (197):174597063, (198):174597067, (199):174597215, (200):174597317, (201):174597325, (202):174597456, (203)
:174597491, (204):174597748, (205):174597790, (206):174597812, (207):174597854, (208):174597948, (209):174597969, (210):174598023, (211):174598027, (212):174598149, (213):174598202, (214):174598237, (215):174598254, (216):174598455, (217):174598459, (218):174598481, (219):1745985
24, (220):174598565, (221):174598588, (222):174598691, (223):174598757, (224):174598802, (225):174592782, (226):174592898, (227):174592987, (228):174593087, (229):174593101, (230):174593133, (231):174593228, (232):174593274, (233):174593398, (234):174593419, (235):174593473, (236
):174593498, (237):174593552, (238):174593583, (239):174593589, (240):174593612, (241):174593627, (242):174593606, (243):174593734, (244):174593906, (245):174593926, (246):174594018, (247):174594087, (248):174594147, (249):174594175, (250):174594258, (251):174594288, (252):174594
347, (253):174594428, (254):174594500, (255):174594588, (256):174594623, (257):174594651, (258):174594666, (259):174594734, (260):174594788, (261):174594802, (262):174595005, (263):174595079, (264):174595160, (265):174595202, (266):174595210, (267):174595288, (268):174595313, (26
9):174595317, (270):174595414, (271):174595422, (272):174595456, (273-275):174595595-174595597, (276):174595671, (277):174595811, (278):174595816, (279):174595891, (280):174595921, (281):174595957, (282-283):174595968-174595969, (284):174596028, (285):174596100, (286):174596133,
(287):174596214, (288):174596247, (289-293):174596304-174596308, (294):174596357, (295):174596395, (296):174596426, (297):174596454, (298):174596464, (299):174596469, (300):174596558, (301):174596569, (302):174596580, (303-305):174596594-174596596, (306):174596621, (307):17459664
5, (308):174596649, (309):174596735, (310):174596760, (311):174584603, (312):174584680, (313):174584713, (314):174584815, (315):174584902, (316):174584940, (317):174584957, (318):174585040, (319):174585055, (320):174585089, (321):174585183, (322):174585199, (323-324):174585256-17
4585257, (325):174585338, (326):174585350, (327):174585441, (328):174585494, (329):174585775, (330):174585801, (331):174585917, (332):174585933, (333):174585962, (334):174585984, (335):174586019, (336):174586062, (337):174586067, (338):174586112, (339):174586128, (340):174586162,
(341):174586241, (342):174586265, (343):174586396, (344):174586498, (345):174586701, (346):174586730, (347):174586749, (348):174586791, (349):174586795, (350):174586835, (351):174586841, (352):174586936, (353):174587019, (354):174587195, (355):174587204, (356):174587233, (357):1
74587262, (358):174587440, (359):174587594, (360):174587641, (361):174587659, (362):174587682, (363):174587693, (364):174587721, (365):174587766, (366):174587782, (367):174587798, (368):174587808, (369):174587843, (370):174588009, (371):174588069, (372):174588073, (373):174588076
, (374):174588153, (375):174588182, (376):174588197, (377):174588264, (378):174588302, (379):174588368, (380):174588379, (381):174588525, (382):174588579, (383):174588646, (384):174588668, (385):174588709, (386):174588712, (387):174588788, (388):174588803, (389):174588864, (390):
174588872, (391):174588920, (392):174588979, (393):174589067, (394):174589117, (395):174589321, (396):174589338, (397):174589533, (398):174589567, (399):174589746, (400):174589766, (401):174589811, (402):174589824, (403):174589880, (404):174589955, (405):174589967, (406):17459003
2, (407):174590146, (408):174590165, (409):174590174, (410):174590317, (411):174590407, (412):174590678, (413):174590734, (414):174590742, (415):174590834, (416):174590921, (417):174590994, (418):174591117, (419):174591146, (420):174591279, (421):174591314, (422):174591339, (423)
:174591396, (424):174591401, (425):174591408, (426):174591462, (427):174591550, (428):174591594, (429):174591670, (430):174591678, (431):174591757, (432):174591783, (433):174591785, (434):174590526, (435):174590549, (436):174591968, (437):174592085, (438):174592123, (439):1745921
75, (440):174592252, (441):174592302, (442):174592308, (443):174592327, (444):174592373, (445):174592385, (446):174592439, (447):174592454, (448):174592464, (449):174592523, (450):174592531, (451):174592632, (452):174617277, (453):174617174, (454):174617058, (455):174617078, (456
):174616802, (457):174616820, (458):174616836, (459):174616944, (460):174616418, (461):174616492, (462):174616496, (463):174616655, (464):174616665, (465):174616675, (466):174616677, (467):174616690, (468):174615273, (469):174615282, (470):174615354, (471):174615361, (472):174615
394, (473):174615397, (474):174615413, (475):174615506, (476):174615673, (477):174615677, (478):174615690, (479):174615693, (480):174615791, (481):174615800, (482):174615802, (483):174615884, (484):174615893, (485):174615995, (486):174616056, (487):174616072, (488):174616152, (48
9):174616193, (490):174613252, (491):174613271, (492):174613390, (493):174613485, (494):174613495, (495):174613579, (496):174613644, (497):174613651, (498):174613665, (499):174613674, (500):174613693, (501):174613696, (502):174613851, (503):174613858, (504):174613880, (505-506):1
74614020-174614021, (507):174614136, (508):174614141, (509):174614182, (510):174614339, (511):174614406, (512):174614830, (513-514):174614937-174614938, (515):174615166, (516):174609142, (517):174609167, (518):174609228, (519):174609418, (520):174609497, (521):174609535, (522):17
4609604, (523):174609644, (524):174609661, (525):174609692, (526):174609714, (527):174609744, (528):174609759, (529):174609875, (530):174609884, (531):174609890, (532):174610000, (533):174610160, (534):174610446, (535):174610529, (536):174610542, (537):174610552, (538):174610734,
(539):174610754, (540):174610856, (541):174610883, (542):174610937, (543):174611057, (544):174611102, (545):174611183, (546):174611207, (547):174611223, (548):174611267, (549):174611333, (550):174611351, (551):174611541, (552):174611552, (553):174611586, (554):174611614, (555):1
74611670, (556):174611698, (557):174611715, (558):174611900, (559):174611974, (560):174612003, (561):174612026, (562):174612156, (563):174612163, (564):174612168, (565):174612240, (566):174612318, (567):174612335, (568):174612373, (569):174612389, (570):174612518, (571):174612523
, (572):174612539, (573):174612599, (574):174612707, (575):174612711, (576):174612768, (577):174612812, (578):174612901, (579):174612914, (580):174612937, (581):174613034, (582):174600969, (583):174601025, (584):174601031, (585):174601153, (586):174601172, (587):174601260, (588):
174601267, (589):174601282, (590):174601285, (591):174601314, (592):174601403, (593):174601446, (594):174601457, (595):174601486, (596):174601584, (597):174601658, (598):174601769, (599):174601876, (600):174601918, (601):174601952, (602):174602039, (603):174602065, (604):17460208
7, (605):174602098, (606):174602115, (607):174602130, (608):174602244, (609):174602343, (610):174602397, (611):174602444, (612):174602488, (613):174602507, (614):174602571, (615):174602823, (616):174602836, (617):174602942, (618):174602962, (619):174602990, (620):174603001, (621)
:174603028, (622):174603054, (623):174603082, (624):174603084, (625):174603152, (626):174603289, (627):174603342, (628):174603429, (629):174603539, (630):174603589, (631):174603635, (632):174603645, (633):174603700, (634):174603814, (635):174603819, (636):174603852, (637):1746039
55, (638):174603965, (639):174604000, (640):174604022, (641):174604067, (642):174604124, (643):174604149, (644):174604152, (645):174604327, (646):174604488, (647):174604569, (648):174604587, (649):174604600, (650):174604658, (651):174604671, (652):174604704, (653):174604721, (654
):174604773, (655):174604781, (656):174604868, (657):174604944, (658):174604949, (659):174604954, (660):174605002, (661):174605062, (662):174605085, (663):174605215, (664):174605261, (665):174605271, (666):174605305, (667):174605449, (668):174605579, (669):174605714, (670):174605
785, (671):174605808, (672):174605869, (673):174605902, (674):174605910, (675):174605978, (676):174606001, (677):174606029, (678):174606056, (679):174606273, (680):174606588, (681):174606596, (682):174606608, (683):174606626, (684):174606658, (685):174606722, (686):174606763, (68
7):174606817, (688):174606824, (689):174606907, (690):174606945, (691):174607011, (692):174607159, (693):174607217, (694):174607223, (695):174607423, (696):174607435, (697):174607493, (698):174607534, (699):174607588, (700):174607740, (701):174607845, (702):174607872, (703):17460
7904, (704):174607907, (705):174607925, (706):174607964, (707):174607971, (708):174608039, (709):174608041, (710):174608203, (711):174608220, (712):174608240, (713):174608278, (714):174608321, (715):174608351, (716):174608408, (717):174608469, (718):174608471, (719):174608547, (7
20):174608555, (721):174608868, (722):174608878, (723):174608949, (724):174608957, (725):174608994, (726):174609024, (727):174633582, (728):174633516, (729):174633225, (730):174633234, (731):174633293, (732):174632650, (733):174632709, (734):174632726, (735):174632736, (736):1746
32749, (737):174632832, (738):174632889, (739):174632932, (740):174632981, (741):174631655, (742):174631679, (743):174631774, (744):174631785, (745):174631794, (746):174631991, (747):174632025, (748):174632040, (749):174632079, (750):174632081, (751):174632099, (752):174632124, (
753):174632131, (754):174632261, (755):174632430, (756):174632493, (757):174632496, (758):174632539, (759):174632596, (760):174632604, (761):174629579, (762):174629658, (763):174629692, (764):174629732, (765):174629889, (766):174629953, (767):174630112, (768):174630115, (769):174
630385, (770):174630436, (771):174630444, (772):174630766, (773):174630803, (774):174630833, (775):174630836, (776):174630850, (777):174630959, (778):174630978, (779):174631118, (780):174631158, (781):174631222, (782):174631286, (783):174631327, (784):174631331, (785):174631383,
(786):174631439, (787):174631504, (788):174631518, (789):174631524, (790):174631585, (791):174631611, (792):174625559, (793):174625606, (794):174625708, (795):174625752, (796):174625790, (797):174625804, (798):174625865, (799):174625920, (800):174626105, (801):174626141, (802):17
4626258, (803):174626363, (804):174626378, (805):174626388, (806):174626428, (807):174626487, (808):174626495, (809):174626555, (810):174626589, (811):174626670, (812):174626711, (813):174626734, (814):174626851, (815):174626908, (816):174626959, (817):174627028, (818):174627039,
(819):174627045, (820):174627059, (821):174627072, (822):174627194, (823):174627198, (824):174627421, (825):174627466, (826):174627485, (827):174627537, (828):174627553, (829):174627637, (830):174627648, (831):174627671, (832):174627692, (833):174627757, (834):174627765, (835):1
74627774, (836):174627798, (837):174628038, (838):174628066, (839):174628224, (840):174628343, (841):174628477, (842):174628614, (843):174628816, (844):174628821, (845):174628823, (846):174628963, (847):174628986, (848):174629020, (849):174629162, (850):174629170, (851):174629191
, (852):174629207, (853):174629230, (854):174629363, (855):174629369, (856):174629406, (857):174629420, (858):174629471, (859):174617305, (860):174617366, (861):174617391, (862):174617455, (863):174617493, (864):174617521, (865):174617648, (866):174617661, (867):174617751, (868):
174617812, (869):174617880, (870):174617940, (871):174617972, (872):174618018, (873):174618075, (874):174618082, (875):174618109, (876):174618115, (877):174618154, (878):174618181, (879):174618193, (880):174618209, (881):174618216, (882):174618223, (883):174618244, (884):17461831
9, (885):174618331, (886):174618333, (887):174618358, (888):174618388, (889):174618418, (890):174618479, (891):174618551, (892):174618564, (893):174618590, (894):174618625, (895):174618680, (896):174618770, (897):174618902, (898):174619001, (899):174619057, (900):174619093, (901)
:174619138, (902):174619181, (903):174619188, (904):174619259, (905):174619294, (906):174619431, (907):174619498, (908):174619558, (909):174619616, (910):174619681, (911):174619691, (912):174619717, (913):174619767, (914):174619787, (915):174619814, (916):174619872, (917):1746199
25, (918):174619970, (919):174620045, (920):174620165, (921):174620182, (922):174620203, (923):174620217, (924):174620271, (925):174620326, (926):174620819, (927):174620853, (928):174620879, (929):174620896, (930):174620917, (931):174620925, (932):174620964, (933):174621083, (934
):174621121, (935):174621155, (936):174621185, (937):174621304, (938):174621353, (939):174621578, (940):174621597, (941):174621781, (942):174621819, (943):174621877, (944):174621889, (945):174621950, (946):174622007, (947):174622116, (948):174622350, (949):174622384, (950):174622
506, (951):174622694, (952):174622814, (953):174622843, (954):174622854, (955):174622901, (956):174622945, (957):174622956, (958):174623120, (959):174623147, (960):174623558, (961):174623565, (962):174623601, (963):174623633, (964):174623709, (965):174623721, (966):174623724, (96
7):174623868, (968):174623924, (969):174623953, (970):174623977, (971):174624023, (972):174624025, (973):174624066, (974):174624262, (975):174624305, (976):174624316, (977):174624479, (978):174624506, (979):174624509, (980):174624518, (981):174624579, (982):174624593, (983):17462
4697, (984):174624737, (985):174624808, (986):174624818, (987):174624916, (988):174625017, (989):174625056, (990):174625117, (991):174625304, (992):174625355, (993):174625362, (994):174625372, (995):174625408, (996):174625415, (997):174625478, (998):174650032, (999):174649858, (1
000):174649888, (1001):174649583, (1002):174649589, (1003):174649602, (1004):174649079, (1005):174649081, (1006):174649112, (1007):174649161, (1008):174649179, (1009):174649294, (1010):174649340, (1011):174649353, (1012):174649405, (1013):174648056, (1014):174648092, (1015):17464
8115, (1016):174648183, (1017):174648213, (1018):174648243, (1019):174648354, (1020):174648453, (1021-1022):174648584-174648585, (1023):174648615, (1024):174648631, (1025):174648647, (1026):174648674, (1027):174648686, (1028):174648693, (1029):174648703, (1030):174648739, (1031-1
032):174648779-174648780, (1033):174648909, (1034):174648959, (1035):174645961, (DIND):174646027, (IND):174646028, (1036):174646029, (1037):174646061, (1038):174646128, (1039):174646167, (1040):174646194, (1041):174646290, (1042):174646440, (1043):174646470, (1044):174646696, (10
45):174646727, (1046):174646783, (1047):174646805, (1048):174646928, (1049):174646944, (1050):174647006, (1051):174647088, (1052):174647106, (1053):174647167, (1054):174647339, (1055):174647571, (1056):174647577, (1057):174647602, (1058):174647614, (1059):174647623, (1060):174647
641, (1061):174647745, (1062):174647766, (1063):174647808, (1064):174647813, (1065):174647842, (1066):174647859, (1067):174647887, (1068):174647968, (1069):174647992, (1070):174641867, (1071):174641900, (1072):174641915, (1073):174641987, (1074):174642078, (1075):174642087, (1076
):174642122, (1077):174642128, (1078):174642218, (1079):174642255, (1080):174642302, (1081):174642359, (1082):174642399, (1083):174642414, (1084):174642518, (1085):174642560, (1086):174642562, (1087):174642570, (1088):174642682, (1089):174642755, (1090):174642758, (1091):17464277
3, (1092):174642841, (1093):174642890, (1094):174642918, (1095):174642975, (1096):174643040, (1097):174643045, (1098):174643123, (1099):174643264, (1100):174643297, (1101):174643332, (1102):174643362, (1103):174643443, (1104):174643500, (1105):174643609, (1106):174643765, (1107):
174643804, (1108):174643864, (1109):174643882, (1110):174643889, (1111):174644019, (1112):174644026, (1113):174644137, (1114):174644255, (1115):174644292, (1116):174644440, (1117):174644444, (1118):174644460, (1119):174644685, (1120):174644745, (1121):174644853, (1122):174644855,
(1123):174644884, (1124):174644942, (1125):174645030, (1126):174645054, (1127):174645063, (1128):174645140, (1129):174645168, (1130):174645319, (1131):174645327, (1132):174645430, (1133):174645432, (1134):174645436, (1135):174645440, (1136):174645448, (1137):174645499, (1138):17
4645634, (1139):174645660, (1140):174645729, (1141):174645899, (1142):174633695, (1143):174633710, (1144):174633841, (1145):174633911, (1146):174633928, (1147):174634094, (1148):174634126, (1149):174634151, (1150):174634176, (1151):174634264, (1152):174634304, (1153):174634347, (
1154):174634401, (1155):174634403, (1156):174634466, (1157):174634485, (1158):174634574, (1159):174634592, (1160):174634662, (1161):174634863, (1162):174634972, (1163):174634999, (1164):174635006, (1165):174635113, (1166):174635137, (1167):174635159, (1168):174635174, (1169):1746
35289, (1170):174635368, (1171):174635370, (1172):174635385, (1173):174635426, (1174):174635544, (1175):174635562, (1176):174635581, (1177):174635680, (1178):174635696, (1179):174635798, (1180):174635812, (1181):174635821, (1182):174635879, (1183):174635938, (1184):174635971, (11
85):174636069, (1186):174636132, (1187):174636263, (1188):174636281, (1189):174636286, (1190):174636313, (1191):174636362, (1192):174636416, (1193):174636448, (1194):174636724, (1195):174636758, (1196):174636792, (1197):174636908, (1198):174637103, (1199):174637294, (1200):174637
329, (1201):174637346, (1202):174637536, (1203):174637608, (1204):174637630, (1205):174637649, (1206):174637657, (1207):174637669, (1208):174637735, (1209):174637815, (1210):174637910, (1211):174638114, (1212):174638171, (1213):174638185, (1214):174638348, (1215):174638461, (1216
):174638489, (1217):174638593, (1218):174638598, (1219):174638641, (1220):174638646, (1221):174638661, (1222):174638686, (1223):174638783, (1224):174638946, (1225):174638948, (1226):174638982, (1227):174639034, (1228):174639102, (1229):174639106, (1230):174639120, (1231):17463929
0, (1232):174639317, (1233):174639489, (1234):174639636, (1235):174639645, (1236):174639867, (1237):174639921, (1238):174639997, (1239):174640037, (1240):174640149, (1241):174640207, (1242):174640221, (1243):174640269, (1244):174640392, (1245):174640474, (1246):174640576, (1247):
174640610, (1248):174640624, (1249):174640628, (1250):174640683, (1251):174640772, (1252):174640805, (1253):174640812, (1254):174640874, (1255):174640932, (1256):174640963, (1257):174640989, (1258):174641051, (1259):174641095, (1260):174641169, (1261):174641181, (1262):174641197,
(1263):174641251, (1264):174641253, (1265):174641293, (1266):174641391, (1267):174641441, (1268):174641461, (1269):174641480, (1270):174641526, (1271):174641534, (1272):174641563, (1273):174641568, (1274):174641630, (1275):174641658, (1276):174641694, (1277):174641699, (1278):17
4641771, (1279):174641833, (1280):174641839, (1281):174666407, (1282):174666384, (1283):174666300, (1284):174666325, (1285):174666343, (1286):174666182, (1287):174665964, (1288):174666147, (1289):174665514, (1290):174665537, (1291):174665803, (1292):174665806, (1293):174665864, (
1294):174664394, (1295):174664405, (1296):174664420, (1297):174664431, (1298):174664534, (1299):174664539, (1300):174664578, (1301):174664641, (1302):174664650, (1303):174664671, (1304):174664694, (1305):174664758, (1306):174664880, (1307):174664905, (1308):174664939, (1309):1746
65042, (1310):174665054, (1311):174665056, (1312):174665079, (1313):174665138, (1314):174665240, (1315):174665250, (1316):174665354, (1317):174662351, (1318):174662359, (1319):174662433, (1320):174662459, (1321):174662475, (1322):174662562, (1323):174662596, (1324):174662674, (13
25):174662704, (1326):174662717, (1327):174662843, (1328):174662918, (1329):174663011, (1330):174663043, (1331):174663107, (1332):174663152, (1333):174663238, (1334):174663416, (1335):174663446, (1336):174663636, (1337):174663639, (1338):174663665, (1339):174663693, (1340):174663
713, (1341):174663716, (1342):174663735, (1343):174663819, (1344):174663856, (1345):174663935, (1346):174664009, (1347):174664052, (1348):174664163, (1349):174664210, (1350):174664212, (1351):174664231, (1352):174664356, (1353):174658334, (1354):174658451, (1355):174658506, (1356
):174658520, (1357):174658590, (1358):174658594, (1359):174658621, (1360):174658639, (1361):174658678, (1362):174658717, (1363):174658719, (1364):174658774, (1365):174658893, (1366):174658921, (1367):174658930, (1368):174658956, (1369):174658962, (1370):174658991, (1371):17465906
1, (1372):174659099, (1373):174659127, (1374):174659276, (1375):174659440, (1376):174659508, (1377):174659545, (1378):174659631, (1379):174659719, (1380):174659798, (1381):174659821, (1382):174659912, (1383):174659944, (1384):174660036, (1385):174660051, (1386):174660083, (1387):
174660107, (1388):174660187, (1389):174660319, (1390):174660355, (1391):174660450, (1392):174660571, (1393):174660629, (1394):174660700, (1395):174660721, (1396):174661025, (1397):174661061, (1398):174661114, (1399):174661182, (1400):174661219, (1401):174661221, (1402):174661261,
(1403):174661551, (1404):174661619, (1405):174661636, (1406):174661663, (1407):174661773, (1408):174661788, (1409):174661825, (1410):174661891, (1411):174661906, (1412):174661927, (1413):174662008, (1414):174662011, (1415):174662017, (1416):174662094, (1417):174662166, (1418):17
4662287, (1419):174662291, (1420):174650135, (1421):174650160, (1422):174650177, (1423):174650203, (1424):174650278, (1425):174650383, (1426):174650588, (1427):174650642, (1428):174650684, (1429):174650762, (1430):174650838, (1431):174650900, (1432):174650986, (1433):174651051, (
1434):174651055, (1435):174651083, (1436):174651134, (1437):174651260, (1438):174651326, (1439):174651452, (1440):174651555, (1441):174651567, (1442):174651642, (1443):174651766, (1444):174651768, (1445):174651873, (1446):174651930, (1447):174651949, (1448):174652067, (1449):1746
52139, (1450):174652170, (1451):174652177, (1452):174652217, (1453):174652231, (1454):174652306, (1455):174652339, (1456):174652379, (1457):174652397, (1458):174652468, (1459):174652520, (1460):174652547, (1461):174652584, (1462):174652609, (1463):174652647, (1464):174652692, (14
65):174652697, (1466):174652809, (1467):174652829, (1468):174653048, (1469):174653114, (1470):174653279, (1471):174653342, (1472):174653366, (1473):174653482, (1474):174653563, (1475):174653610, (1476):174653625, (1477):174653650, (1478):174653678, (1479):174653772, (1480):174653
799, (1481):174653865, (1482):174653875, (1483):174653961, (1484):174654046, (1485):174654151, (1486):174654180, (1487):174654210, (1488):174654247, (1489):174654293, (1490):174654391, (1491):174654406, (1492):174654519, (1493):174654603, (1494):174654669, (1495):174654692, (1496
):174654788, (1497):174654879, (1498):174654909, (1499):174654921, (1500):174654925, (1501):174654984, (1502):174655262, (1503):174655276, (1504):174655344, (1505):174655361, (1506):174655366, (1507):174655377, (1508):174655380, (1509):174655422, (1510):174655625, (1511):17465577
1, (1512):174655787, (1513):174655818, (1514):174655922, (1515):174655952, (1516):174655989, (1517):174656135, (1518):174656139, (1519):174656191, (1520):174656302, (1521):174656355, (1522):174656411, (1523):174656441, (1524):174656492, (1525):174656537, (1526):174656558, (1527):
174656600, (1528):174656640, (1529):174656688, (1530):174656847, (1531):174656937, (1532):174656949, (1533):174656962, (1534):174657031, (1535):174657082, (1536):174657125, (1537):174657142, (1538):174657213, (1539):174657234, (1540):174657245, (1541):174657416, (1542):174657429,
(1543):174657460, (1544):174657568, (1545):174657649, (1546):174657662, (1547):174657691, (1548):174657693, (1549):174657727, (1550):174657794, (1551):174657883, (1552):174657983, (1553):174657997, (1554):174658037, (1555):174658083, (1556):174658161, (1557):174658177, (1558):17
4658184, (1559):174658206, (1560):174682786, (1561):174682776, (1562):174682755, (1563):174682584, (1564):174682614, (1565):174682647, (1566):174682334, (1567):174682426, (1568):174682435, (1569):174682437, (1570):174682508, (1571):174681866, (1572):174681909, (1573):174681914, (
1574):174681953, (1575):174681962, (1576):174682009, (1577):174682093, (1578):174682101, (1579):174682138, (1580):174680760, (1581):174680780, (1582):174680784, (1583):174680950, (1584):174681050, (1585):174681059, (1586):174681107, (1587):174681167, (1588):174681231, (1589):1746
81240, (1590):174681299, (1591):174681343, (1592):174681360, (1593):174681385, (1594):174681491, (1595):174681498, (1596):174681540, (1597):174681544, (1598):174681561, (1599):174681573, (1600):174681651, (1601):174681715, (1602):174681774, (1603):174678730, (1604):174678769, (16
05):174678780, (1606):174678805, (1607):174678885, (1608):174678907, (1609):174678912, (1610):174679056, (1611):174679071, (1612):174679096, (1613):174679102, (1614):174679143, (1615):174679300, (1616):174679302, (1617):174679310, (1618):174679394, (1619):174679514, (1620):174679
553, (1621):174679579, (1622):174679602, (1623):174679616, (1624):174679706, (1625):174679823, (1626):174679826, (1627):174679836, (1628):174679838, (1629):174679920, (1630):174679928, (1631):174679967, (1632):174680068, (1633):174680073, (1634):174680135, (1635):174680219, (1636
):174680269, (1637):174680320, (1638):174680369, (1639):174680682, (1640):174680722, (1641):174680751, (1642):174674795, (1643):174674849, (1644):174674866, (1645):174674877, (1646):174674899, (1647):174674917, (1648):174674920, (1649):174674956, (1650):174675027, (1651):17467508
3, (1652):174675123, (1653):174675170, (1654):174675205, (1655):174675212, (1656):174675227, (1657):174675360, (1658):174675366, (1659):174675421, (1660):174675508, (1661):174675588, (1662):174675634, (1663):174675651, (1664):174675723, (1665):174675768, (1666):174675845, (1667):
174675864, (1668):174675871, (1669):174675933, (1670):174675948, (1671):174676151, (1672):174676153, (1673):174676204, (1674):174676263, (1675):174676274, (1676):174676300, (1677):174676327, (1678):174676414, (1679):174676425, (1680):174676440, (1681):174676508, (1682):174676614,
(1683):174676662, (1684):174676700, (1685):174676703, (1686):174676727, (1687):174676771, (1688):174676805, (1689):174676852, (1690):174676875, (1691):174676908, (1692):174677063, (1693):174677073, (1694):174677122, (1695):174677130, (1696):174677156, (1697):174677205, (1698):17
4677226, (1699):174677241, (1700):174677315, (1701):174677532, (1702):174677597, (1703):174677921, (1704):174677930, (1705):174677951
TOTAL: 1709
|
| Comment by Andreas Dilger [ 06/Nov/18 ] |
|
I'd guess that this is a very old filesystem? Do you know with which Lustre version it was first formatted? The files in question all date from Nov 2013, which puts it days after 2.5 released, or more likely some 2.4.x release. The MDT inode FID sequence numbers are very large (not that this is a problem in itself, the sequence numbers are 64-bit values and you've used only 600M of them, but just a confirmation of my guess). The "trusted.fid" xattr on the OSTs that stores a backpointer with the FID of the MDT inode that is using the object has grown over time, as file layouts became more complex (e.g. stripe_size, stripe_count, component_<start,end>, component_id for PFL layouts). The trusted.fid xattrs for the OST objects look to be valid from the debugfs stat output, and are correctly referencing the parent FID and storing the stripe count. The OI Scrub code should upgrade old pre-2.5 trusted.fid to store the parent FID into the trusted.lma xattr, so that it would be consistent between MDTs and OSTs, but it seems like OI Scrub was never run on some of the OSTs since 2.5 was released? I see two different forms of fid xattr on the objects: ------ service213 115 ----- Inode: 19432703 Type: regular Mode: 0666 Flags: 0x80000 Generation: 3229129909 Version: 0x00000000:01a4c126 User: 9440 Group: 26209 Size: 1048576 File ACL: 0 Links: 1 Blockcount: 2048 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x5bb7c237:00000000 -- Fri Oct 5 12:57:43 2018 atime: 0x527b011c:00000000 -- Wed Nov 6 18:55:24 2013 mtime: 0x527b0183:00000000 -- Wed Nov 6 18:57:07 2013 crtime: 0x527b011c:3ec25968 -- Wed Nov 6 18:55:24 2013 Size of extra inode fields: 28 Extended attributes: trusted.fid (32) = 4c 9d 17 24 02 00 00 00 83 28 00 00 38 00 00 00 a1 5d 22 00 00 00 00 00 00 00 00 00 00 00 00 00 fid: objid=2252193 seq=0 parent=[0x224179d4c:0x2883:0x0] stripe=56 EXTENTS: (0-255):5782965760-5782966015 ------ service205 185 ----- Inode: 1183292 Type: regular Mode: 0666 Flags: 0x80000 Generation: 990691313 Version: 0x00000000:01aaf3cf User: 9440 Group: 26209 Size: 883968 File ACL: 0 Links: 1 Blockcount: 1728 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x5bb7c237:00000000 -- Fri Oct 5 12:57:43 2018 atime: 0x527b0110:00000000 -- Wed Nov 6 18:55:12 2013 mtime: 0x527b0183:00000000 -- Wed Nov 6 18:57:07 2013 crtime: 0x527b0110:d09557f4 -- Wed Nov 6 18:55:12 2013 Size of extra inode fields: 28 Extended attributes: trusted.fid (32) = 4c 9d 17 24 02 00 00 00 83 28 00 00 39 00 00 00 f4 33 22 00 00 00 00 00 00 00 00 00 00 00 00 00 fid: objid=2241524 seq=0 parent=[0x224179d4c:0x2883:0x0] stripe=57 EXTENTS: (0-215):340487973-340488188 ------ service212 166 ----- Inode: 12104635 Type: regular Mode: 0666 Flags: 0x80000 Generation: 1266395486 Version: 0x00000000:00000000 User: 9440 Group: 26209 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x5bb7c237:00000000 -- Fri Oct 5 12:57:43 2018 atime: 0x527b00fc:642516b8 -- Wed Nov 6 18:54:52 2013 mtime: 0x527b00fc:642516b8 -- Wed Nov 6 18:54:52 2013 crtime: 0x527b00fc:642516b8 -- Wed Nov 6 18:54:52 2013 Size of extra inode fields: 28 Extended attributes: trusted.fid (16) = 4c 9d 17 24 02 00 00 00 83 28 00 00 3a 00 00 00 fid: parent=[0x224179d4c:0x2883:0x0] stripe=58 ------ service216 66 ----- Inode: 7209791 Type: regular Mode: 0666 Flags: 0x80000 Generation: 3980417589 Version: 0x00000000:00000000 User: 9440 Group: 26209 Size: 0 File ACL: 0 Links: 1 Blockcount: 0 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x5bb7c237:00000000 -- Fri Oct 5 12:57:43 2018 atime: 0x527b00b1:c4503b20 -- Wed Nov 6 18:53:37 2013 mtime: 0x527b00b1:c4503b20 -- Wed Nov 6 18:53:37 2013 crtime: 0x527b00b1:c4503b20 -- Wed Nov 6 18:53:37 2013 Size of extra inode fields: 28 Extended attributes: trusted.fid (16) = 4c 9d 17 24 02 00 00 00 83 28 00 00 9f 00 00 00 fid: parent=[0x224179d4c:0x2883:0x0] stripe=159 As you can see, there are trusted.fid xattrs with 16-byte size, and some with 32-bite size (the old format that also stored the objid and seq values of the object itself. Looking at this particular file, however, it seems like this old file is striped over 160 OSTs and has a file size of 60652800 bytes, so it is not using all of the OST object, but seems otherwise intact. The client starts accessing the file but hits an error: : : 00020000:00000002:10.0:1541487278.411671:0:96313:0:(lov_offset.c:251:lov_stripe_intersects()) [0->-1] -> [(-1) 0->-1 (0)] 00020000:00000002:10.0:1541487278.411690:0:96313:0:(lov_offset.c:251:lov_stripe_intersects()) [0->-1] -> [(-1) 0->-1 (0)] 00000100:02020000:18.0:1541487278.412587:0:7535:0:(client.c:1266:ptlrpc_check_status()) 11-0: nbp8-OST00a6-osc-ffff881f0ecc2800: operation ldlm_enqueue to node 10.151.27.72@o2ib failed: rc = -22 00000100:02020000:6.0:1541487278.412613:0:7537:0:(client.c:1266:ptlrpc_check_status()) 11-0: nbp8-OST0054-osc-ffff881f0ecc2800: operation ldlm_enqueue to node 10.151.27.68@o2ib failed: rc = -22 00000100:02020000:26.0:1541487278.412627:0:7536:0:(client.c:1266:ptlrpc_check_status()) 11-0: nbp8-OST00ab-osc-ffff881f0ecc2800: operation ldlm_enqueue to node 10.151.27.77@o2ib failed: rc = -22 : : 00000100:02020000:2.0:1541487278.415811:0:7524:0:(client.c:1266:ptlrpc_check_status()) 11-0: nbp8-OST0042-osc-ffff881f0ecc2800: operation ldlm_enqueue to node 10.151.27.76@o2ib failed: rc = -22 for OST indices 166, 84, 171, ..., 66. According to the file layout these are valid stripes in the file: : : : :
115 2252193 0x225da1 0
185 2241524 0x2233f4 0
166 2231839 0x220e1f 0
171 2230066 0x220732 0
84 2233835 0x2215eb 0
: : : :
66 2234372 0x221804 0
that follow immediately after the last used objects in the file objid=2252193 and objid=2241524 as seen in the debugfs OST object stat above. What it looks like is that the OST objects which were actually storing data for the file had their "trusted.fid" xattrs updated properly to store their "self" FID, but objects beyond the end of the file do not store their "self" FID unless they are actually modified by a client. Can you please capture Lustre kernel debug logs from an OSS with lctl set_param debug=-1 set while accessing some of these files. I'd like to see where the -22 = -EINVAL error is coming from, so that we can understand what needs to be fixed. As for solving this, one option would be to downgrade the MDS and OSS temporarily to 2.7.x and rsync the files in this directory to new files, so that they will have new-format xattrs, and then delete the old files. The other option is to temporarily quarantine these files until we can patch the OST so that the objects are accessible, and separately to make an OI Scrub/LFSCK patch to repair the objects. The latter should be done at some point anyway, so that the OST objects can store their "self" FID in the LMA in case there is ever filesystem corruption in the future that dumps the objects into lost+found and they need to be recovered. |
| Comment by Mahmoud Hanafi [ 06/Nov/18 ] |
|
Attaching s219.dk.out.gz The client is 10.151.26.25@o2ib. Looks like osd_check_lma() returning -22 00080000:00000001:5.0:1541523220.922495:0:10991:0:(osd_handler.c:517:osd_iget_check()) Process entered 00080000:00000001:5.0:1541523220.922495:0:10991:0:(osd_handler.c:646:osd_iget_check()) Process leaving via put (rc=0 : 0 : 0x0) 00080000:00000001:5.0:1541523220.922496:0:10991:0:(osd_handler.c:728:osd_check_lma()) Process entered 00080000:00000001:5.0:1541523220.922500:0:10991:0:(osd_handler.c:748:osd_check_lma()) Process leaving (rc=18446744073709551594 : -22 : ffffffffffffffea) 00080000:00000001:5.0:1541523220.922501:0:10991:0:(osd_handler.c:1201:osd_fid_lookup()) Process leaving via out (rc=18446744073709551594 : -22 : 0xffffffffffffffea)
|
| Comment by Gerrit Updater [ 08/Nov/18 ] |
|
Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/33627 |
| Comment by Mahmoud Hanafi [ 09/Nov/18 ] |
|
Can we get a 2.10.5 back port please. |
| Comment by Peter Jones [ 09/Nov/18 ] |
|
Mahmoud Again, are you wanting this ahead of the patch being finalized? Peter |
| Comment by Mahmoud Hanafi [ 09/Nov/18 ] |
|
Yes I think so. Yesterday Andreas said it is low risk. We run just to copy the files and then revert. |
| Comment by Andreas Dilger [ 12/Nov/18 ] |
|
I was going to port this to b2_10 but I see that the master version of this patch has failed sanity-scrub with problems that look related to the patch, so I don't think it is safe to put into production. I was en route to SC today, but I'll try to look at it later tonight. |
| Comment by Mahmoud Hanafi [ 25/Nov/18 ] |
|
Any upates?
|
| Comment by Mahmoud Hanafi [ 04/Dec/18 ] |
|
If the patch going to take some time, do we need to go back to 2.7.x or can we go back to 2.9 copy the files. |
| Comment by Andreas Dilger [ 05/Dec/18 ] |
|
Mahmoud, sorry for the slow reply on this, we've migrated mail servers and I missed your previous comment. As for downgrading, I would not necessarily recommend going back to 2.9 if you upgraded from 2.7.x. You might consider to downgrade to 2.7.x for a short time in a maintenance window to copy the files and then return to 2.10.x again. If any files have been created with 2.10 features (e.g. PFL composite layouts) then those files will not be accessible under 2.7.x, so it should be OK to downgrade for a short time but I wouldn't recommend it for an extended uptime given that you have been running 2.10.x for some time already. Also, given that there are still some latent problems in the filesystem the fewer files that are accessed/modified while running 2.7.x the better. |
| Comment by Mahmoud Hanafi [ 05/Dec/18 ] |
|
What about running with the patch for a short time (maintenance window). Going back to 2.7.x would require downgrading the OS to cent6.x but if we can run with the patch for a short time it may be a short maintenance. |
| Comment by Mahmoud Hanafi [ 13/Dec/18 ] |
|
Downgrade to 2.7.3 couldn't mount MDT 00000001:00020000:15.0:1544742465.414368:0:6712:0:(tgt_lastrcvd.c:1072:tgt_server_data_init()) nbp8-MDT0000: unsupported incompat filesystem feature(s) 400 I need to try again when I am able to umount all clients. I think a patch would be better. What about a lfsck patch to update the objects. |
| Comment by Peter Jones [ 17/Dec/18 ] |
|
Alex Could you please assist with this? Thanks Peter |
| Comment by Alex Zhuravlev [ 18/Dec/18 ] |
|
/** multiple RPCs in flight //* multiple RPCs in flight */ #define OBD_INCOMPAT_MULTI_RPCS 0x00000400 iirc, this flag is reset when all clients are unmounted. is it possible to start newer MDS, unmount the client (or abort recovery to get all the clients evicted) ?
|
| Comment by Mahmoud Hanafi [ 18/Dec/18 ] |
|
I did try abort recovery but it didn't work. Unless we take a cluster wide dedicated time it will be difficult to cleanly umount all clients. So I think the patch will be a better options for us. How much rework does the patch require to be consider safe.
|
| Comment by Gerrit Updater [ 21/Dec/18 ] |
|
Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/33906 |
| Comment by Alex Zhuravlev [ 21/Dec/18 ] |
|
mhanafi please don't use this patch yet - it's still under testing. |
| Comment by Mahmoud Hanafi [ 26/Dec/18 ] |
|
Has there been any additional progress on this patch. |
| Comment by Gerrit Updater [ 04/Jan/19 ] |
|
Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/33958 |
| Comment by Mahmoud Hanafi [ 15/Jan/19 ] |
|
Please provide an update on the status of the patch. |
| Comment by Peter Jones [ 21/Jan/19 ] |
|
Mahmoud The patch is undergoing pre-landing testing ATM so hopefully will land in the coming days. Peter |
| Comment by Gerrit Updater [ 23/Jan/19 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33627/ |
| Comment by Peter Jones [ 23/Jan/19 ] |
|
Landed for 2.13 |
| Comment by Mahmoud Hanafi [ 24/Jan/19 ] |
|
Is the patch required on the client side? Or can this fix work work older clients? |
| Comment by Alex Zhuravlev [ 27/Jan/19 ] |
|
mhanafi no, you don't need to change the clients. the idea is that the server upgrades LOV EA properly. |
| Comment by Gerrit Updater [ 15/Feb/19 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/33958/ |
| Comment by Gerrit Updater [ 25/Feb/19 ] |
|
Minh Diep (mdiep@whamcloud.com) uploaded a new patch: https://review.whamcloud.com/34304 |
| Comment by Gerrit Updater [ 19/Mar/19 ] |
|
Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/34304/ |
