Large directory feature is not enabled on this filesystem <4>[401750.854147] Lustre: atlas1-MDT0000: Client f61fdcb8-c0b3-942c-f229-3feb74233a5e (at 1809@gni100) reconnecting <4>[401750.865800] Lustre: Skipped 1 previous similar message <6>[401750.871886] Lustre: atlas1-MDT0000: Connection restored to f61fdcb8-c0b3-942c-f229-3feb74233a5e (at 1809@gni100) <6>[401750.883823] Lustre: Skipped 8 previous similar messages <4>[401753.420890] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401753.434706] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401757.030334] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401757.044163] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401757.682484] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401757.701637] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401764.907104] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401764.920910] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401765.927520] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401765.941331] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401767.983714] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401767.997532] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401770.069663] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401770.083483] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401771.816079] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401771.829883] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401775.393572] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401775.407419] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401778.248960] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401778.262771] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401778.549514] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401778.563323] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401779.843743] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401779.857537] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401780.561912] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401780.575760] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401784.880487] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401784.894269] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401784.961678] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401784.975507] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401786.430631] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401786.444477] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401786.769414] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401786.783218] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401790.328206] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401790.342010] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401793.054058] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401793.067906] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401797.011191] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401797.025019] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401801.367870] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401801.381674] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401801.587929] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401801.601750] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401804.683076] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401804.696880] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401805.194333] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401805.208142] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401808.518234] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401808.532123] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401815.239816] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401815.253623] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401817.602098] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401817.615905] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401825.345327] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401825.359129] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401826.137486] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401826.151331] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401827.031887] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401827.045713] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401828.129060] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401828.142860] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401831.769351] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401831.783175] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401832.725497] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401832.739306] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401837.724128] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401837.737956] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401840.891714] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401840.905528] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401843.227074] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401843.240931] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401844.439578] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401844.453447] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401845.604593] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401845.618420] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401847.140264] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401847.154109] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401853.391181] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401853.405028] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401854.817464] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401854.831322] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401857.336527] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401857.350359] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401857.363990] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401857.377792] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401863.755888] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401863.769751] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401864.544057] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401864.557870] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401864.851381] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401864.865191] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401866.180389] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401866.194192] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401866.552159] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401866.571210] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401874.339973] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401874.353778] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401877.237733] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401877.251546] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401879.712736] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401879.726558] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401880.578207] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401880.592026] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401881.303002] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401881.316813] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401881.757767] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401881.771579] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401882.870084] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401882.883912] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401882.936853] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401882.950670] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401886.120433] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401886.134255] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401890.638927] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401890.652756] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401891.531904] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401891.545717] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401892.258109] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401892.271903] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401897.896988] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401897.910790] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401901.400759] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401901.414563] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401906.988503] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401907.002345] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401907.692930] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401907.706741] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401908.528968] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401908.542779] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401913.831396] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401913.845186] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401915.267281] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401915.281096] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401915.750911] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401915.764710] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401915.845648] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401915.859459] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401917.455615] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401917.469452] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401922.205089] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401922.218908] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401922.464644] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401922.478471] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401922.713505] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401922.727331] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401924.672589] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401924.686380] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401926.528496] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401926.542305] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401927.918661] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401927.932508] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401928.968206] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401928.982054] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401929.262789] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401929.276614] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401930.408493] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401930.422298] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401932.839016] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401932.852819] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401933.253626] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401933.267426] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401935.409106] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401935.422909] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401937.147251] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401937.161040] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401939.315739] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401939.329603] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401942.815834] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401942.829664] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401948.380194] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401948.394030] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401955.292393] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401955.306214] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401956.830999] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401956.844935] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401960.325633] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401960.339486] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401960.456959] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401960.470792] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401961.967639] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401961.981452] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401963.034370] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401963.048203] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401964.927686] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401964.946789] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401966.189597] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401966.203418] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401966.217498] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401966.231312] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401968.396395] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401968.410193] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401974.960235] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401974.974071] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401978.258709] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401978.272555] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401983.154316] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401983.168138] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401983.787733] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401983.801606] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401985.717886] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401985.731685] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401990.016970] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401990.030925] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401990.250909] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401990.264719] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401991.302839] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401991.316648] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401995.285480] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401995.299304] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401996.691429] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401996.705278] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[401999.756407] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[401999.770241] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402000.304263] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402000.318095] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402003.192790] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402003.206596] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402014.066263] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402014.080093] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402016.046678] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402016.060511] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402016.895788] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402016.909725] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402017.764326] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402017.778262] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402019.593143] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402019.606986] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402020.742045] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402020.755860] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402021.768896] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402021.782722] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402022.194066] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402022.207911] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402029.067360] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402029.081182] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402029.731445] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402029.745263] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402031.048997] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402031.062800] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402036.759745] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402036.773587] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402041.276420] Lustre: atlas1-MDT0000: Client 2b49b5a4-48aa-96d1-bba0-0df60f413181 (at 1800@gni100) reconnecting <6>[402041.280354] Lustre: atlas1-MDT0000: Connection restored to e700e7ef-9941-888a-05ce-eb6bbd6b8095 (at 3347@gni100) <6>[402041.280356] Lustre: Skipped 4 previous similar messages <4>[402041.306143] Lustre: Skipped 7 previous similar messages <4>[402041.347335] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402041.361170] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402052.754113] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402052.767938] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402055.039077] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402055.052882] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402057.819002] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402057.832833] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402063.516057] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402063.529863] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402064.328760] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402064.342561] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402064.963646] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402064.977449] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402066.956422] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402066.970229] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402067.072196] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402067.086010] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402075.002302] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402075.016206] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402075.090380] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402075.104189] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402077.453362] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402077.467182] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402092.017069] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402092.030908] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402092.984021] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402092.997811] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402095.857469] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402095.871268] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402095.889787] Lustre: 15940:0:(osd_handler.c:473:osd_ldiskfs_add_entry()) atlas1-MDT0000: directory (inode: 438307289 FID: [0x2003a006c:0x3:0x0]) has reached maximum entry limit <4>[402095.908046] Lustre: 15940:0:(osd_handler.c:473:osd_ldiskfs_add_entry()) Skipped 273 previous similar messages <4>[402096.814943] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402096.828765] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402101.516008] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402101.529813] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402105.237624] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402105.251427] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402106.505752] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402106.519577] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402107.984284] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402107.998127] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402113.022886] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402113.036709] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402119.220909] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402119.234717] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402121.490545] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402121.504359] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402131.076640] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402131.090486] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402134.537142] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402134.550956] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402135.832819] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402135.846640] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402140.042569] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402140.056384] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402147.672011] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402147.685823] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402156.631355] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402156.645190] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402163.313191] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402163.327003] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402163.463235] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402163.477034] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402164.116239] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402164.130053] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402167.854368] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402167.868245] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402173.435928] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402173.449759] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402174.059172] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402174.072982] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402175.332249] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402175.346030] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402175.925645] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402175.939437] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402178.707280] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402178.721083] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402180.479306] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402180.493128] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402186.932602] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402186.946409] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402188.298601] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402188.312411] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402190.055924] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402190.069720] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402190.756500] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402190.770310] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402193.219611] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402193.233438] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402195.198579] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402195.212382] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402203.571314] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402203.585120] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402205.597140] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402205.610964] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402207.688368] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402207.702204] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402209.689387] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402209.703206] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402220.972148] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402220.985963] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402227.053099] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402227.066893] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402237.465632] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402237.479476] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402245.014331] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402245.028140] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402248.156902] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402248.170720] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402255.199514] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402255.213332] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402264.038780] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402264.052617] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402266.964976] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402266.978806] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402268.104549] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402268.118399] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402270.059667] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402270.078802] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402272.822916] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402272.836721] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402275.830997] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402275.844818] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402285.386995] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402285.400850] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402286.627783] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402286.641600] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402304.767021] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402304.780825] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402308.341684] Lustre: atlas1-MDT0000: Client 6b96ea3a-f263-57fa-d5f3-6bcb0addf3a0 (at 2788@gni100) reconnecting <4>[402308.353326] Lustre: Skipped 3 previous similar messages <6>[402308.359573] Lustre: atlas1-MDT0000: Connection restored to 6b96ea3a-f263-57fa-d5f3-6bcb0addf3a0 (at 2788@gni100) <6>[402308.371496] Lustre: Skipped 6 previous similar messages <4>[402310.260233] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402310.274068] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402324.592424] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402324.606244] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402328.021893] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402328.035718] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402334.661185] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402334.675016] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402342.608958] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402342.622777] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402351.641638] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402351.655488] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402356.578295] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402356.592134] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402356.785662] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402356.799491] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402360.379260] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402360.393071] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402374.679691] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402374.693549] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402391.542837] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402391.556648] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402395.948570] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402395.962385] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402398.426384] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402398.440176] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402418.357966] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402418.371773] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402435.051187] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402435.065000] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402437.581632] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402437.595452] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402471.253296] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402471.267116] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402481.218963] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402481.232851] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402484.661265] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402484.675089] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402488.679349] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402488.693170] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402504.472151] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402504.485979] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402517.847438] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402517.861287] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402520.224274] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402520.238110] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402544.476902] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402544.490730] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402570.149717] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402570.163510] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402609.073589] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402609.087421] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402615.683697] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402615.697515] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402617.421444] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402617.435254] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402617.906886] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402617.920700] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402652.504558] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402652.518402] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402678.483349] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402678.497137] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402695.607788] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402695.621580] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402728.739897] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402728.753708] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402728.766941] Lustre: 15870:0:(osd_handler.c:473:osd_ldiskfs_add_entry()) atlas1-MDT0000: directory (inode: 438307289 FID: [0x2003a006c:0x3:0x0]) has reached maximum entry limit <4>[402728.785201] Lustre: 15870:0:(osd_handler.c:473:osd_ldiskfs_add_entry()) Skipped 81 previous similar messages <4>[402786.710493] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402786.724405] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402827.769127] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402827.783010] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402832.115893] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402832.129725] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402921.522208] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402921.536026] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402954.342619] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402954.356428] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[402962.972749] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[402962.986595] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[403058.486757] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[403058.500623] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[403063.310588] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[403063.324370] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[403096.734960] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[403096.748755] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[403106.072793] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[403106.086582] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[403138.132732] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[403138.146543] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[403221.922672] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[403221.936493] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[403249.432441] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[403249.446296] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[403315.892924] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[403315.906728] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[403458.176058] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Directory (ino: 438307289) index full, reach max htree level :2 <4>[403458.189871] LDISKFS-fs warning (device dm-5): ldiskfs_dx_add_entry: Large directory feature is not enabled on this filesystem <4>[403458.203107] Lustre: 15492:0:(osd_handler.c:473:osd_ldiskfs_add_entry()) atlas1-MDT0000: directory (inode: 438307289 FID: [0x2003a006c:0x3:0x0]) has reached maximum entry limit <4>[403458.221401] Lustre: 15492:0:(osd_handler.c:473:osd_ldiskfs_add_entry()) Skipped 14 previous similar messages <6>[407596.121158] Lustre: atlas1-MDT0000: Connection restored to 913dc0a0-1f18-18dc-46ec-843fbfa8f2b5 (at 0@lo) <6>[407596.132443] Lustre: Skipped 2 previous similar messages <6>[407596.138740] Lustre: client wants to enable acl, but mdt not! <4>[407596.326221] Lustre: Mounted atlas1-client <4>[408845.185914] swapper: page allocation failure. order:1, mode:0x20 <4>[408845.192949] Pid: 0, comm: swapper Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[408845.201781] Call Trace: <4>[408845.204831] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[408845.213375] [] ? cpumask_next_and+0x29/0x50 <4>[408845.220215] [] ? kmem_getpages+0x62/0x170 <4>[408845.226864] [] ? fallback_alloc+0x1ba/0x270 <4>[408845.233707] [] ? cache_grow+0x2cf/0x320 <4>[408845.240160] [] ? ____cache_alloc_node+0x99/0x160 <4>[408845.247489] [] ? kmem_cache_alloc_node_trace+0x90/0x200 <4>[408845.255498] [] ? __kmalloc_node+0x4d/0x60 <4>[408845.262143] [] ? __alloc_skb+0x7a/0x190 <4>[408845.268595] [] ? dev_alloc_skb+0x1d/0x40 <4>[408845.275180] [] ? ipoib_alloc_rx_skb+0x3f/0x200 [ib_ipoib] <4>[408845.283386] [] ? mlx4_ib_poll_cq+0x52a/0xd30 [mlx4_ib] <4>[408845.291300] [] ? ipoib_ib_handle_rx_wc+0x8c/0x300 [ib_ipoib] <4>[408845.300035] [] ? ipoib_poll+0x14b/0x180 [ib_ipoib] <4>[408845.307550] [] ? net_rx_action+0x103/0x300 <4>[408845.314308] [] ? __do_softirq+0xe5/0x230 <4>[408845.320868] [] ? mlx4_msi_x_interrupt+0x14/0x20 [mlx4_core] <4>[408845.329503] [] ? call_softirq+0x1c/0x30 <4>[408845.335967] [] ? do_softirq+0x65/0xa0 <4>[408845.342220] [] ? irq_exit+0x85/0x90 <4>[408845.348291] [] ? do_IRQ+0x75/0xf0 <4>[408845.354183] [] ? ret_from_intr+0x0/0x11 <4>[408845.360647] [] ? intel_idle+0xfe/0x1b0 <4>[408845.367704] [] ? intel_idle+0xe1/0x1b0 <4>[408845.374057] [] ? sched_clock+0x9/0x10 <4>[408845.380326] [] ? sched_clock_cpu+0xcd/0x110 <4>[408845.387167] [] ? cpuidle_idle_call+0x7a/0xe0 <4>[408845.394105] [] ? cpu_idle+0xb6/0x110 <4>[408845.400266] [] ? start_secondary+0x2c0/0x316 <4>[410329.648638] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[410339.956100] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[410537.766160] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[410877.866121] Lustre: atlas1-MDT0000: Client f8ad7a26-75f2-711f-fc63-c4367ace2198 (at 15020@gni100) reconnecting <4>[410877.877848] Lustre: Skipped 2 previous similar messages <6>[410877.884034] Lustre: atlas1-MDT0000: Connection restored to f8ad7a26-75f2-711f-fc63-c4367ace2198 (at 15020@gni100) <6>[410877.896071] Lustre: Skipped 1 previous similar message <6>[410879.712855] Lustre: atlas1-MDT0000: Connection restored to 0e035273-a60a-84a6-6344-5829a15114c7 (at 15765@gni100) <6>[410879.724873] Lustre: Skipped 2 previous similar messages <4>[410885.058267] Lustre: atlas1-MDT0000: Client d4284072-d738-b8c5-c864-f3da6cf822fb (at 16570@gni100) reconnecting <4>[410885.069999] Lustre: Skipped 5 previous similar messages <6>[410885.076169] Lustre: atlas1-MDT0000: Connection restored to d4284072-d738-b8c5-c864-f3da6cf822fb (at 16570@gni100) <6>[410885.088181] Lustre: Skipped 1 previous similar message <6>[410889.465622] Lustre: atlas1-MDT0000: Connection restored to 9309c5bc-2b95-1d5d-3302-842b85f8e575 (at 14157@gni100) <6>[410889.477639] Lustre: Skipped 3 previous similar messages <4>[410899.062167] Lustre: atlas1-MDT0000: Client f0a2e979-7475-a6c5-428f-4e207171fa48 (at 15046@gni100) reconnecting <4>[410899.073895] Lustre: Skipped 4 previous similar messages <6>[410899.080070] Lustre: atlas1-MDT0000: Connection restored to f0a2e979-7475-a6c5-428f-4e207171fa48 (at 15046@gni100) <4>[411177.079524] Lustre: atlas1-MDT0000: Client f8ad7a26-75f2-711f-fc63-c4367ace2198 (at 15020@gni100) reconnecting <6>[411177.083024] Lustre: atlas1-MDT0000: Connection restored to f014c3c3-051a-f2c6-0d51-66d859aec4b9 (at 14234@gni100) <4>[411177.103319] Lustre: Skipped 6 previous similar messages <4>[413408.429770] Lustre: Unmounted atlas1-client <4>[421003.924630] Lustre: atlas1-MDT0000: haven't heard from client 58d0bb45-c0cb-2aa1-3a2c-91ed3e11c1c7 (at 15285@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0b3a6000, cur 1486916105 expire 1486915205 last 1486914753 <4>[424713.353590] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[424725.780215] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[424925.499423] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[431898.910763] Lustre: atlas1-MDT0000: haven't heard from client 72ed39ba-b3ee-2ad4-f766-d3163eb5e547 (at 5536@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f109f0400, cur 1486927000 expire 1486926100 last 1486925648 <4>[438929.874258] Lustre: 14923:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1486933176/real 1486933176] req@ffff880b03827980 x1558706251404384/t0(0) o6->atlas1-OST0223-osc-MDT0000@10.36.225.145@o2ib:28/4 lens 664/432 e 12 to 1 dl 1486934026 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[438929.907609] Lustre: 14923:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 27 previous similar messages <4>[438929.919135] Lustre: atlas1-OST0223-osc-MDT0000: Connection to atlas1-OST0223 (at 10.36.225.145@o2ib) was lost; in progress operations using this service will wait for recovery to complete <6>[438930.018298] Lustre: atlas1-OST0223-osc-MDT0000: Connection restored to 10.36.225.145@o2ib (at 10.36.225.145@o2ib) <6>[438930.030307] Lustre: Skipped 6 previous similar messages <4>[439000.465102] Lustre: 14920:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1486933176/real 1486933176] req@ffff881cc50976c0 x1558706251417120/t0(0) o6->atlas1-OST00b8-osc-MDT0000@10.36.225.70@o2ib:28/4 lens 664/432 e 12 to 1 dl 1486934026 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[439000.498192] Lustre: atlas1-OST00b8-osc-MDT0000: Connection to atlas1-OST00b8 (at 10.36.225.70@o2ib) was lost; in progress operations using this service will wait for recovery to complete <6>[439000.594157] Lustre: atlas1-OST00b8-osc-MDT0000: Connection restored to 10.36.225.70@o2ib (at 10.36.225.70@o2ib) <4>[439137.313870] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[439147.732986] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[439205.490863] Lustre: 14922:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1486933176/real 1486933176] req@ffff8819d2b7acc0 x1558706251395608/t0(0) o6->atlas1-OST0368-osc-MDT0000@10.36.225.38@o2ib:28/4 lens 664/432 e 12 to 1 dl 1486934026 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[439205.523966] Lustre: atlas1-OST0368-osc-MDT0000: Connection to atlas1-OST0368 (at 10.36.225.38@o2ib) was lost; in progress operations using this service will wait for recovery to complete <6>[439205.621627] Lustre: atlas1-OST0354-osc-MDT0000: Connection restored to 10.36.225.162@o2ib (at 10.36.225.162@o2ib) <4>[439359.681825] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[439418.900997] Lustre: atlas1-MDT0000: haven't heard from client b4f84673-8c26-f1af-0eca-ec75b8893fa0 (at 17816@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f11b0c800, cur 1486934520 expire 1486933620 last 1486933168 <4>[451528.883590] Lustre: atlas1-MDT0000: haven't heard from client 76e83216-b663-bdcd-3c99-eef0e7fd2f5b (at 16467@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f10075c00, cur 1486946630 expire 1486945730 last 1486945278 <4>[461725.452194] Lustre: atlas1-MDT0000: Client e0aef344-cbc7-99df-a57d-c4e62188b53d (at 10440@gni100) reconnecting <6>[461725.469210] Lustre: atlas1-MDT0000: Connection restored to e0aef344-cbc7-99df-a57d-c4e62188b53d (at 10440@gni100) <6>[461725.481252] Lustre: Skipped 1 previous similar message <4>[461732.237679] Lustre: atlas1-MDT0000: Client 32a73e00-cdfe-fb35-98e8-4c81437bb26a (at 11947@gni100) reconnecting <4>[461732.249442] Lustre: Skipped 2 previous similar messages <6>[461732.255664] Lustre: atlas1-MDT0000: Connection restored to 32a73e00-cdfe-fb35-98e8-4c81437bb26a (at 11947@gni100) <6>[461732.267702] Lustre: Skipped 2 previous similar messages <6>[461735.485644] Lustre: atlas1-MDT0000: Connection restored to 9c7dd390-b8fc-ce57-d27e-fe4e07d1e7f8 (at 10433@gni100) <4>[462024.696023] Lustre: atlas1-MDT0000: Client 4f7d46c8-b192-132c-8bf6-f1d1021e54b1 (at 10428@gni100) reconnecting <4>[462024.707748] Lustre: Skipped 1 previous similar message <6>[462024.713886] Lustre: atlas1-MDT0000: Connection restored to 4f7d46c8-b192-132c-8bf6-f1d1021e54b1 (at 10428@gni100) <4>[462298.981598] Lustre: atlas1-MDT0000: Client 8d81bb68-8b03-3675-8ee9-532e0d4575e7 (at 11042@gni100) reconnecting <4>[462298.993329] Lustre: Skipped 1 previous similar message <6>[462298.999476] Lustre: atlas1-MDT0000: Connection restored to 8d81bb68-8b03-3675-8ee9-532e0d4575e7 (at 11042@gni100) <6>[462299.011493] Lustre: Skipped 1 previous similar message <4>[496723.252839] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[496733.703273] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[496939.624291] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[498509.819007] Lustre: atlas1-MDT0000: haven't heard from client 2af8b941-bc0d-5ec1-b3f4-061fa557c854 (at 18223@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f101ad000, cur 1486993611 expire 1486992711 last 1486992259 <4>[499685.817191] Lustre: atlas1-MDT0000: haven't heard from client e27b6c2a-e24f-a62b-d618-27bf961154d3 (at 2296@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0d818400, cur 1486994787 expire 1486993887 last 1486993435 <6>[502813.491558] Lustre: atlas1-MDT0000: Connection restored to 5f24f235-86e7-2c97-5b0f-7f9fe0b5cf62 (at 17752@gni100) <6>[502813.503587] Lustre: Skipped 1 previous similar message <6>[502815.687616] Lustre: atlas1-MDT0000: Connection restored to 6801b0da-fa36-f81a-9a2a-552f01d6aed1 (at 2392@gni100) <6>[502815.699537] Lustre: Skipped 23 previous similar messages <6>[502819.827565] Lustre: atlas1-MDT0000: Connection restored to ec49f5de-2b7e-9893-f1f6-f7cf7e8b1ef7 (at 17578@gni100) <6>[502819.839613] Lustre: Skipped 41 previous similar messages <6>[502827.862391] Lustre: atlas1-MDT0000: Connection restored to 69dfef15-603f-08cb-c366-8e4200645c5d (at 17658@gni100) <6>[502827.874415] Lustre: Skipped 82 previous similar messages <4>[502988.813181] Lustre: atlas1-MDT0000: haven't heard from client ebc60729-356b-8b06-9012-dbe2e952e30f (at 14956@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f12277000, cur 1486998090 expire 1486997190 last 1486996738 <4>[502988.838554] Lustre: Skipped 219 previous similar messages <4>[503439.813051] Lustre: atlas1-MDT0000: haven't heard from client 117a5783-bf23-5fe4-a5ff-e34cbc35fc8b (at 18320@gni100) in 1348 seconds. I think it's dead, and I am evicting it. exp ffff883f12484400, cur 1486998541 expire 1486997641 last 1486997193 <4>[507079.807169] Lustre: atlas1-MDT0000: haven't heard from client d0985963-769b-4483-db36-324d357e0de3 (at 10.39.232.84@o2ib6) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883ed789a800, cur 1487002181 expire 1487001281 last 1487000829 <4>[507079.833133] Lustre: Skipped 215 previous similar messages <4>[509438.804288] Lustre: atlas1-MDT0000: haven't heard from client 6b553cac-9862-1cad-f0a0-ccebcb48e751 (at 16320@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883edc179800, cur 1487004540 expire 1487003640 last 1487003188 <6>[510729.605352] Lustre: atlas1-MDT0000: Connection restored to 4923572e-ed91-5b96-b55a-a6b3bd832150 (at 10.36.247.168@o2ib) <6>[510729.618041] Lustre: Skipped 70 previous similar messages <4>[511117.227119] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[511127.664264] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[511385.427838] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <6>[516176.213859] Lustre: atlas1-MDT0000: Connection restored to b192b184-43d9-64ca-fdb3-b2e46b694552 (at 10.36.247.141@o2ib) <6>[517287.397390] Lustre: atlas1-MDT0000: Connection restored to 1a49502e-fbfa-32b9-10ce-f052d1c2eebe (at 10.36.247.131@o2ib) <6>[517305.291473] Lustre: atlas1-MDT0000: Connection restored to 2b026413-36f0-236a-44ed-de91ddef693f (at 10.36.247.132@o2ib) <6>[518197.172475] Lustre: atlas1-MDT0000: Connection restored to e7d8345c-c098-84b3-9f33-ba1a35d22643 (at 17@gni4) <6>[518197.184008] Lustre: Skipped 25 previous similar messages <6>[520329.780074] Lustre: atlas1-MDT0000: Connection restored to 37a74511-34a1-3b44-46f5-c98b8be2cdfb (at 17004@gni100) <6>[520330.294103] Lustre: atlas1-MDT0000: Connection restored to 7db2de00-0f9f-3c41-fd84-c4945c5dd596 (at 96@gni100) <6>[520330.305872] Lustre: Skipped 7 previous similar messages <6>[520331.600824] Lustre: atlas1-MDT0000: Connection restored to a168185e-b318-1739-7954-94c13af194e0 (at 18332@gni100) <6>[520331.612893] Lustre: Skipped 22 previous similar messages <6>[520333.808980] Lustre: atlas1-MDT0000: Connection restored to 1b98f65c-ce53-3969-9f62-5dd7adecbe49 (at 18320@gni100) <6>[520333.820997] Lustre: Skipped 9 previous similar messages <6>[520337.959835] Lustre: atlas1-MDT0000: Connection restored to 77779fbc-039e-17d9-8e8a-817fd44986fb (at 110@gni100) <6>[520337.971668] Lustre: Skipped 43 previous similar messages <6>[520345.961453] Lustre: atlas1-MDT0000: Connection restored to 2fa84987-6873-2255-a315-4aedce98d6f6 (at 18617@gni100) <6>[520345.973470] Lustre: Skipped 86 previous similar messages <6>[520728.106076] Lustre: atlas1-MDT0000: Connection restored to 7462ee4c-eda2-4f65-7419-7e623514bebd (at 2960@gni100) <6>[520728.118012] Lustre: Skipped 39 previous similar messages <6>[522454.310876] Lustre: atlas1-MDT0000: Connection restored to fc030fb4-8505-a147-96b2-bcba2bfea3e7 (at 16994@gni100) <6>[522454.322895] Lustre: Skipped 82 previous similar messages <6>[522460.109350] Lustre: atlas1-MDT0000: Connection restored to e44c0cd6-591f-1e2a-4cf7-d4b52d6a64f0 (at 16995@gni100) <6>[522471.208817] Lustre: atlas1-MDT0000: Connection restored to 6a0819b8-2c0f-321a-ccbd-6cad89f1f48a (at 17085@gni100) <6>[522471.220848] Lustre: Skipped 1 previous similar message <4>[522567.786377] Lustre: atlas1-MDT0000: haven't heard from client 2c80145b-655e-60c0-681b-da3c6ec650e9 (at 22@gni4) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883ed1d8b400, cur 1487017669 expire 1487016769 last 1487016317 <6>[522836.312243] Lustre: atlas1-MDT0000: Connection restored to 93718e62-1111-4efa-56ec-e8e85abfdf46 (at 17084@gni100) <3>[523524.502273] LustreError: 15562:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20039f31f:0xf4ab:0x0]: rc = -2 <4>[523609.950194] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x1000:0x15c5020:0x0] with flags 0x4a, rc = 0 <4>[523934.126690] Lustre: atlas1-MDT0000: Client 51a4b9e5-35d8-d569-ef69-948d39272fce (at 10.36.225.6@o2ib) reconnecting <4>[523934.138841] Lustre: Skipped 1 previous similar message <6>[523934.146214] Lustre: atlas1-MDT0000: Connection restored to 51a4b9e5-35d8-d569-ef69-948d39272fce (at 10.36.225.6@o2ib) <4>[524189.058094] rdma_cm: page allocation failure. order:5, mode:0xd0 <4>[524189.065140] Pid: 2777, comm: rdma_cm Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[524189.074247] Call Trace: <4>[524189.077309] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[524189.084959] [] ? alloc_vmap_area+0x27a/0x390 <4>[524189.091930] [] ? kmem_getpages+0x62/0x170 <4>[524189.098584] [] ? fallback_alloc+0x1ba/0x270 <4>[524189.105433] [] ? cache_grow+0x2cf/0x320 <4>[524189.111911] [] ? ____cache_alloc_node+0x99/0x160 <4>[524189.119278] [] ? create_qp_common+0xedb/0xf50 [mlx4_ib] <4>[524189.127310] [] ? __kmalloc+0x199/0x230 <4>[524189.133697] [] ? create_qp_common+0xedb/0xf50 [mlx4_ib] <4>[524189.141715] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[524189.149361] [] ? mlx4_ib_create_qp+0x166/0x2b0 [mlx4_ib] <4>[524189.157486] [] ? ib_create_qp+0x41/0x1c0 [ib_core] <4>[524189.165042] [] ? rdma_create_qp+0x48/0xc0 [rdma_cm] <4>[524189.172689] [] ? kiblnd_create_conn+0xa1d/0x16e0 [ko2iblnd] <4>[524189.181305] [] ? kiblnd_cm_callback+0x124f/0x2090 [ko2iblnd] <4>[524189.190043] [] ? cma_work_handler+0x7c/0xb0 [rdma_cm] <4>[524189.197865] [] ? cma_work_handler+0x0/0xb0 [rdma_cm] <4>[524189.205599] [] ? worker_thread+0x170/0x2a0 <4>[524189.212350] [] ? autoremove_wake_function+0x0/0x40 <4>[524189.219886] [] ? worker_thread+0x0/0x2a0 <4>[524189.226439] [] ? kthread+0x9e/0xc0 <4>[524189.232434] [] ? child_rip+0xa/0x20 <4>[524189.238548] [] ? kthread+0x0/0xc0 <4>[524189.244427] [] ? child_rip+0x0/0x20 <6>[524189.250493] Mem-Info: <4>[524189.253370] Node 0 DMA per-cpu: <4>[524189.257229] CPU 0: hi: 0, btch: 1 usd: 0 <4>[524189.262926] CPU 1: hi: 0, btch: 1 usd: 0 <4>[524189.268597] CPU 2: hi: 0, btch: 1 usd: 0 <4>[524189.274292] CPU 3: hi: 0, btch: 1 usd: 0 <4>[524189.279974] CPU 4: hi: 0, btch: 1 usd: 0 <4>[524189.285665] CPU 5: hi: 0, btch: 1 usd: 0 <4>[524189.291351] CPU 6: hi: 0, btch: 1 usd: 0 <4>[524189.297036] CPU 7: hi: 0, btch: 1 usd: 0 <4>[524189.302717] Node 0 DMA32 per-cpu: <4>[524189.306759] CPU 0: hi: 186, btch: 31 usd: 0 <4>[524189.312447] CPU 1: hi: 186, btch: 31 usd: 0 <4>[524189.318132] CPU 2: hi: 186, btch: 31 usd: 0 <4>[524189.323817] CPU 3: hi: 186, btch: 31 usd: 0 <4>[524189.329497] CPU 4: hi: 186, btch: 31 usd: 0 <4>[524189.335176] CPU 5: hi: 186, btch: 31 usd: 0 <4>[524189.340863] CPU 6: hi: 186, btch: 31 usd: 0 <4>[524189.346548] CPU 7: hi: 186, btch: 31 usd: 0 <4>[524189.352223] Node 0 Normal per-cpu: <4>[524189.356368] CPU 0: hi: 186, btch: 31 usd: 178 <4>[524189.362046] CPU 1: hi: 186, btch: 31 usd: 163 <4>[524189.367732] CPU 2: hi: 186, btch: 31 usd: 165 <4>[524189.373409] CPU 3: hi: 186, btch: 31 usd: 0 <4>[524189.379087] CPU 4: hi: 186, btch: 31 usd: 168 <4>[524189.384773] CPU 5: hi: 186, btch: 31 usd: 164 <4>[524189.390451] CPU 6: hi: 186, btch: 31 usd: 180 <4>[524189.396128] CPU 7: hi: 186, btch: 31 usd: 184 <4>[524189.401808] active_anon:84307 inactive_anon:74317 isolated_anon:0 <4>[524189.401808] active_file:26017294 inactive_file:25782419 isolated_file:0 <4>[524189.401808] unevictable:9338 dirty:17399 writeback:0 unstable:0 <4>[524189.401809] free:2316924 slab_reclaimable:9502873 slab_unreclaimable:1468347 <4>[524189.401809] mapped:13286 shmem:136289 pagetables:1517 bounce:0 <4>[524189.444474] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[524189.486444] lowmem_reserve[]: 0 1880 258420 258420 <4>[524189.492223] Node 0 DMA32 free:387068kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[524189.535253] lowmem_reserve[]: 0 0 256540 256540 <4>[524189.540724] Node 0 Normal free:9351464kB min:67088kB low:83860kB high:100632kB active_anon:337228kB inactive_anon:297268kB active_file:103672024kB inactive_file:103121880kB unevictable:37352kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:69596kB writeback:0kB mapped:53144kB shmem:545156kB slab_reclaimable:37930400kB slab_unreclaimable:5872612kB kernel_stack:74784kB pagetables:6068kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[524189.591090] lowmem_reserve[]: 0 0 0 0 <4>[524189.595571] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[524189.608187] Node 0 DMA32: 7*4kB 6*8kB 5*16kB 9*32kB 7*64kB 5*128kB 6*256kB 10*512kB 8*1024kB 5*2048kB 88*4096kB = 387068kB <4>[524189.621279] Node 0 Normal: 2252044*4kB 41308*8kB 5548*16kB 4245*32kB 736*64kB 36*128kB 6*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 9617520kB <4>[524189.636256] 51773584 total pagecache pages <4>[524189.641164] 0 pages in swap cache <4>[524189.645193] Swap cache stats: add 0, delete 0, find 0/0 <4>[524189.651462] Free swap = 0kB <4>[524189.655001] Total swap = 0kB <6>[524190.080338] 67108863 pages RAM <6>[524190.084083] 1015827 pages reserved <6>[524190.088213] 50892254 pages shared <6>[524190.092252] 12334357 pages non-shared <4>[524237.064834] rdma_cm: page allocation failure. order:5, mode:0xd0 <4>[524237.071877] Pid: 2777, comm: rdma_cm Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[524237.081010] Call Trace: <4>[524237.084074] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[524237.091724] [] ? alloc_vmap_area+0x27a/0x390 <4>[524237.098690] [] ? kmem_getpages+0x62/0x170 <4>[524237.105353] [] ? fallback_alloc+0x1ba/0x270 <4>[524237.112223] [] ? cache_grow+0x2cf/0x320 <4>[524237.118695] [] ? ____cache_alloc_node+0x99/0x160 <4>[524237.126055] [] ? create_qp_common+0xedb/0xf50 [mlx4_ib] <4>[524237.134081] [] ? __kmalloc+0x199/0x230 <4>[524237.140453] [] ? create_qp_common+0xedb/0xf50 [mlx4_ib] <4>[524237.148478] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[524237.156120] [] ? mlx4_ib_create_qp+0x166/0x2b0 [mlx4_ib] <4>[524237.164243] [] ? ib_create_qp+0x41/0x1c0 [ib_core] <4>[524237.171790] [] ? rdma_create_qp+0x48/0xc0 [rdma_cm] <4>[524237.179432] [] ? kiblnd_create_conn+0xa1d/0x16e0 [ko2iblnd] <4>[524237.188070] [] ? kiblnd_cm_callback+0x124f/0x2090 [ko2iblnd] <4>[524237.196802] [] ? cma_work_handler+0x7c/0xb0 [rdma_cm] <4>[524237.204638] [] ? cma_work_handler+0x0/0xb0 [rdma_cm] <4>[524237.212370] [] ? worker_thread+0x170/0x2a0 <4>[524237.219138] [] ? autoremove_wake_function+0x0/0x40 <4>[524237.226679] [] ? worker_thread+0x0/0x2a0 <4>[524237.233244] [] ? kthread+0x9e/0xc0 <4>[524237.239226] [] ? child_rip+0xa/0x20 <4>[524237.245302] [] ? kthread+0x0/0xc0 <4>[524237.251184] [] ? child_rip+0x0/0x20 <6>[524237.257267] Mem-Info: <4>[524237.260124] Node 0 DMA per-cpu: <4>[524237.263990] CPU 0: hi: 0, btch: 1 usd: 0 <4>[524237.269677] CPU 1: hi: 0, btch: 1 usd: 0 <4>[524237.275364] CPU 2: hi: 0, btch: 1 usd: 0 <4>[524237.281051] CPU 3: hi: 0, btch: 1 usd: 0 <4>[524237.286733] CPU 4: hi: 0, btch: 1 usd: 0 <4>[524237.292418] CPU 5: hi: 0, btch: 1 usd: 0 <4>[524237.298103] CPU 6: hi: 0, btch: 1 usd: 0 <4>[524237.303792] CPU 7: hi: 0, btch: 1 usd: 0 <4>[524237.309501] Node 0 DMA32 per-cpu: <4>[524237.313562] CPU 0: hi: 186, btch: 31 usd: 0 <4>[524237.319244] CPU 1: hi: 186, btch: 31 usd: 0 <4>[524237.324931] CPU 2: hi: 186, btch: 31 usd: 0 <4>[524237.330612] CPU 3: hi: 186, btch: 31 usd: 0 <4>[524237.336299] CPU 4: hi: 186, btch: 31 usd: 0 <4>[524237.341986] CPU 5: hi: 186, btch: 31 usd: 0 <4>[524237.347675] CPU 6: hi: 186, btch: 31 usd: 0 <4>[524237.353361] CPU 7: hi: 186, btch: 31 usd: 0 <4>[524237.359043] Node 0 Normal per-cpu: <4>[524237.363202] CPU 0: hi: 186, btch: 31 usd: 179 <4>[524237.368876] CPU 1: hi: 186, btch: 31 usd: 26 <4>[524237.374558] CPU 2: hi: 186, btch: 31 usd: 167 <4>[524237.380244] CPU 3: hi: 186, btch: 31 usd: 173 <4>[524237.385927] CPU 4: hi: 186, btch: 31 usd: 63 <4>[524237.391606] CPU 5: hi: 186, btch: 31 usd: 3 <4>[524237.397293] CPU 6: hi: 186, btch: 31 usd: 20 <4>[524237.402978] CPU 7: hi: 186, btch: 31 usd: 156 <4>[524237.408668] active_anon:84422 inactive_anon:74316 isolated_anon:0 <4>[524237.408668] active_file:25740201 inactive_file:25740305 isolated_file:0 <4>[524237.408669] unevictable:9338 dirty:3468 writeback:0 unstable:0 <4>[524237.408669] free:2046577 slab_reclaimable:10100364 slab_unreclaimable:1460694 <4>[524237.408669] mapped:13321 shmem:136295 pagetables:1539 bounce:0 <4>[524237.446201] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[524237.488179] lowmem_reserve[]: 0 1880 258420 258420 <4>[524237.493934] Node 0 DMA32 free:387068kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[524237.536971] lowmem_reserve[]: 0 0 256540 256540 <4>[524237.542443] Node 0 Normal free:8243788kB min:67088kB low:83860kB high:100632kB active_anon:337688kB inactive_anon:297264kB active_file:102774168kB inactive_file:102774980kB unevictable:37352kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:13872kB writeback:0kB mapped:53284kB shmem:545180kB slab_reclaimable:40315320kB slab_unreclaimable:5842388kB kernel_stack:74800kB pagetables:6156kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[524237.592793] lowmem_reserve[]: 0 0 0 0 <4>[524237.597276] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[524237.609978] Node 0 DMA32: 7*4kB 6*8kB 5*16kB 9*32kB 7*64kB 5*128kB 6*256kB 10*512kB 8*1024kB 5*2048kB 88*4096kB = 387068kB <4>[524237.623068] Node 0 Normal: 1997416*4kB 52368*8kB 2947*16kB 1666*32kB 120*64kB 27*128kB 7*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 8522512kB <4>[524237.638038] 51457872 total pagecache pages <4>[524237.642947] 0 pages in swap cache <4>[524237.646980] Swap cache stats: add 0, delete 0, find 0/0 <4>[524237.653153] Free swap = 0kB <4>[524237.656702] Total swap = 0kB <6>[524238.084433] 67108863 pages RAM <6>[524238.088189] 1015827 pages reserved <6>[524238.092340] 50540621 pages shared <6>[524238.096373] 12885830 pages non-shared <6>[524702.870423] Lustre: atlas1-MDT0000: Connection restored to 24852f88-974f-2344-ae0e-3c559d67262f (at 17@gni4) <6>[524704.839038] Lustre: atlas1-MDT0000: Connection restored to a3af9b0b-c641-f1d1-5a88-1f8010986ea1 (at 129@gni4) <4>[525136.844602] Lustre: atlas1-MDT0000: Client a3af9b0b-c641-f1d1-5a88-1f8010986ea1 (at 129@gni4) reconnecting <6>[525136.855945] Lustre: atlas1-MDT0000: Connection restored to a3af9b0b-c641-f1d1-5a88-1f8010986ea1 (at 129@gni4) <4>[525225.026359] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[525228.099150] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[525230.571450] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[525380.747440] Lustre: atlas1-MDT0000: Client 24852f88-974f-2344-ae0e-3c559d67262f (at 17@gni4) reconnecting <6>[525380.758695] Lustre: atlas1-MDT0000: Connection restored to e0988e3d-a9e2-441f-7c5d-cca8e05ebb7d (at 17@gni4) <6>[525441.210614] Lustre: atlas1-MDT0000: Connection restored to b6adb02d-3c10-1ccd-17bd-21deb9a8b10a (at 91@gni4) <6>[525449.364807] Lustre: atlas1-MDT0000: Connection restored to 34d9453c-d9a0-7444-56ab-c0e4c626d3c7 (at 102@gni4) <6>[525449.376442] Lustre: Skipped 151 previous similar messages <6>[525515.178567] Lustre: atlas1-MDT0000: Connection restored to 5ef26c9c-9876-a446-bee7-c481418735e5 (at 120@gni4) <6>[525515.190225] Lustre: Skipped 12 previous similar messages <4>[525536.306319] Lustre: atlas1-MDT0000: Client b6adb02d-3c10-1ccd-17bd-21deb9a8b10a (at 91@gni4) reconnecting <4>[525589.871918] Lustre: atlas1-MDT0000: Client a3af9b0b-c641-f1d1-5a88-1f8010986ea1 (at 129@gni4) reconnecting <4>[525589.883256] Lustre: Skipped 3 previous similar messages <6>[525589.889431] Lustre: atlas1-MDT0000: Connection restored to adf17aa3-aca8-71ac-4375-cca5d6d01a16 (at 129@gni4) <6>[525589.901062] Lustre: Skipped 4 previous similar messages <4>[525762.815784] Lustre: atlas1-MDT0000: Client 24852f88-974f-2344-ae0e-3c559d67262f (at 17@gni4) reconnecting <6>[525762.827048] Lustre: atlas1-MDT0000: Connection restored to 24852f88-974f-2344-ae0e-3c559d67262f (at 17@gni4) <6>[525762.838657] Lustre: Skipped 2 previous similar messages <4>[525799.529674] Lustre: atlas1-MDT0000: Client 09264d1e-8beb-6b5b-5e81-3c7f1b63dba0 (at 77@gni4) reconnecting <4>[529891.320976] dsm_sa_datamgrd: page allocation failure. order:1, mode:0x8020 <4>[529891.329017] Pid: 11764, comm: dsm_sa_datamgrd Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[529891.339027] Call Trace: <4>[529891.342106] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[529891.349735] [] ? try_to_del_timer_sync+0x7b/0xe0 <4>[529891.357096] [] ? dma_generic_alloc_coherent+0xa6/0x160 <4>[529891.365014] [] ? x86_swiotlb_alloc_coherent+0x31/0x70 <4>[529891.372842] [] ? pci_alloc_consistent+0x5a/0xc0 [mpt3sas] <4>[529891.381051] [] ? _ctl_do_mpt_command+0x284/0xcf0 [mpt3sas] <4>[529891.389601] [] ? _ctl_ioctl_main+0x58b/0x1120 [mpt3sas] <4>[529891.397617] [] ? __do_page_fault+0x1f4/0x500 <4>[529891.404560] [] ? _ctl_ioctl+0x16/0x20 [mpt3sas] <4>[529891.411793] [] ? vfs_ioctl+0x22/0xa0 <4>[529891.417957] [] ? do_nanosleep+0x93/0xc0 <4>[529891.424432] [] ? do_vfs_ioctl+0x84/0x580 <4>[529891.430984] [] ? hrtimer_wakeup+0x0/0x30 <4>[529891.437546] [] ? sys_ioctl+0x81/0xa0 <4>[529891.443743] [] ? system_call_fastpath+0x16/0x1b <6>[529891.450984] Mem-Info: <4>[529891.453849] Node 0 DMA per-cpu: <4>[529891.457712] CPU 0: hi: 0, btch: 1 usd: 0 <4>[529891.463386] CPU 1: hi: 0, btch: 1 usd: 0 <4>[529891.469063] CPU 2: hi: 0, btch: 1 usd: 0 <4>[529891.474740] CPU 3: hi: 0, btch: 1 usd: 0 <4>[529891.480416] CPU 4: hi: 0, btch: 1 usd: 0 <4>[529891.486100] CPU 5: hi: 0, btch: 1 usd: 0 <4>[529891.491772] CPU 6: hi: 0, btch: 1 usd: 0 <4>[529891.497445] CPU 7: hi: 0, btch: 1 usd: 0 <4>[529891.503122] Node 0 DMA32 per-cpu: <4>[529891.507177] CPU 0: hi: 186, btch: 31 usd: 0 <4>[529891.512866] CPU 1: hi: 186, btch: 31 usd: 0 <4>[529891.518543] CPU 2: hi: 186, btch: 31 usd: 0 <4>[529891.524225] CPU 3: hi: 186, btch: 31 usd: 0 <4>[529891.529896] CPU 4: hi: 186, btch: 31 usd: 0 <4>[529891.535569] CPU 5: hi: 186, btch: 31 usd: 0 <4>[529891.541263] CPU 6: hi: 186, btch: 31 usd: 0 <4>[529891.546941] CPU 7: hi: 186, btch: 31 usd: 0 <4>[529891.552618] Node 0 Normal per-cpu: <4>[529891.556740] CPU 0: hi: 186, btch: 31 usd: 38 <4>[529891.562419] CPU 1: hi: 186, btch: 31 usd: 192 <4>[529891.568092] CPU 2: hi: 186, btch: 31 usd: 188 <4>[529891.573762] CPU 3: hi: 186, btch: 31 usd: 175 <4>[529891.579436] CPU 4: hi: 186, btch: 31 usd: 170 <4>[529891.585111] CPU 5: hi: 186, btch: 31 usd: 184 <4>[529891.590784] CPU 6: hi: 186, btch: 31 usd: 154 <4>[529891.596466] CPU 7: hi: 186, btch: 31 usd: 113 <4>[529891.602150] active_anon:84571 inactive_anon:74646 isolated_anon:0 <4>[529891.602150] active_file:16222888 inactive_file:16195718 isolated_file:0 <4>[529891.602151] unevictable:9907 dirty:11972 writeback:0 unstable:0 <4>[529891.602151] free:281913 slab_reclaimable:31197704 slab_unreclaimable:1185097 <4>[529891.602152] mapped:11322 shmem:136772 pagetables:1529 bounce:0 <4>[529891.639701] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[529891.681720] lowmem_reserve[]: 0 1880 258420 258420 <4>[529891.687518] Node 0 DMA32 free:387068kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[529891.730581] lowmem_reserve[]: 0 0 256540 256540 <4>[529891.736059] Node 0 Normal free:714924kB min:67088kB low:83860kB high:100632kB active_anon:338284kB inactive_anon:298584kB active_file:64894268kB inactive_file:64783648kB unevictable:39628kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:47888kB writeback:0kB mapped:45288kB shmem:547088kB slab_reclaimable:124797024kB slab_unreclaimable:4740388kB kernel_stack:74752kB pagetables:6116kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[529891.786221] lowmem_reserve[]: 0 0 0 0 <4>[529891.790701] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[529891.803270] Node 0 DMA32: 7*4kB 6*8kB 5*16kB 9*32kB 7*64kB 5*128kB 6*256kB 10*512kB 8*1024kB 5*2048kB 88*4096kB = 387068kB <4>[529891.816377] Node 0 Normal: 173156*4kB 1778*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 706864kB <4>[529891.830086] 32558447 total pagecache pages <4>[529891.834993] 0 pages in swap cache <4>[529891.839015] Swap cache stats: add 0, delete 0, find 0/0 <4>[529891.845185] Free swap = 0kB <4>[529891.848724] Total swap = 0kB <6>[529892.257304] 67108863 pages RAM <6>[529892.261036] 1015827 pages reserved <6>[529892.265154] 32060032 pages shared <6>[529892.269172] 33525046 pages non-shared <4>[530807.306016] dsm_sa_datamgrd: page allocation failure. order:1, mode:0x8020 <4>[530807.314045] Pid: 11764, comm: dsm_sa_datamgrd Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[530807.324019] Call Trace: <4>[530807.327079] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[530807.334704] [] ? try_to_del_timer_sync+0x7b/0xe0 <4>[530807.342032] [] ? dma_generic_alloc_coherent+0xa6/0x160 <4>[530807.349942] [] ? x86_swiotlb_alloc_coherent+0x31/0x70 <4>[530807.357759] [] ? pci_alloc_consistent+0x5a/0xc0 [mpt3sas] <4>[530807.365964] [] ? _ctl_do_mpt_command+0x284/0xcf0 [mpt3sas] <4>[530807.374476] [] ? _ctl_ioctl_main+0x58b/0x1120 [mpt3sas] <4>[530807.382483] [] ? __do_page_fault+0x1f4/0x500 <4>[530807.389424] [] ? _ctl_ioctl+0x16/0x20 [mpt3sas] <4>[530807.396651] [] ? vfs_ioctl+0x22/0xa0 <4>[530807.402812] [] ? do_nanosleep+0x93/0xc0 <4>[530807.409264] [] ? do_vfs_ioctl+0x84/0x580 <4>[530807.415818] [] ? hrtimer_wakeup+0x0/0x30 <4>[530807.422379] [] ? sys_ioctl+0x81/0xa0 <4>[530807.428552] [] ? system_call_fastpath+0x16/0x1b <6>[530807.435780] Mem-Info: <4>[530807.438629] Node 0 DMA per-cpu: <4>[530807.442472] CPU 0: hi: 0, btch: 1 usd: 0 <4>[530807.448146] CPU 1: hi: 0, btch: 1 usd: 0 <4>[530807.453819] CPU 2: hi: 0, btch: 1 usd: 0 <4>[530807.459492] CPU 3: hi: 0, btch: 1 usd: 0 <4>[530807.465196] CPU 4: hi: 0, btch: 1 usd: 0 <4>[530807.470869] CPU 5: hi: 0, btch: 1 usd: 0 <4>[530807.476550] CPU 6: hi: 0, btch: 1 usd: 0 <4>[530807.482224] CPU 7: hi: 0, btch: 1 usd: 0 <4>[530807.487899] Node 0 DMA32 per-cpu: <4>[530807.491934] CPU 0: hi: 186, btch: 31 usd: 0 <4>[530807.497608] CPU 1: hi: 186, btch: 31 usd: 0 <4>[530807.503285] CPU 2: hi: 186, btch: 31 usd: 0 <4>[530807.508996] CPU 3: hi: 186, btch: 31 usd: 0 <4>[530807.514673] CPU 4: hi: 186, btch: 31 usd: 0 <4>[530807.520346] CPU 5: hi: 186, btch: 31 usd: 0 <4>[530807.526046] CPU 6: hi: 186, btch: 31 usd: 0 <4>[530807.531773] CPU 7: hi: 186, btch: 31 usd: 0 <4>[530807.537455] Node 0 Normal per-cpu: <4>[530807.541598] CPU 0: hi: 186, btch: 31 usd: 214 <4>[530807.547303] CPU 1: hi: 186, btch: 31 usd: 224 <4>[530807.553033] CPU 2: hi: 186, btch: 31 usd: 190 <4>[530807.558718] CPU 3: hi: 186, btch: 31 usd: 236 <4>[530807.564399] CPU 4: hi: 186, btch: 31 usd: 204 <4>[530807.570072] CPU 5: hi: 186, btch: 31 usd: 210 <4>[530807.575778] CPU 6: hi: 186, btch: 31 usd: 34 <4>[530807.581463] CPU 7: hi: 186, btch: 31 usd: 181 <4>[530807.587171] active_anon:84612 inactive_anon:74722 isolated_anon:0 <4>[530807.587171] active_file:16377889 inactive_file:16326744 isolated_file:0 <4>[530807.587172] unevictable:9907 dirty:12528 writeback:0 unstable:0 <4>[530807.587172] free:166633 slab_reclaimable:31059537 slab_unreclaimable:1152459 <4>[530807.587173] mapped:11256 shmem:136870 pagetables:1529 bounce:0 <4>[530807.624645] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[530807.666613] lowmem_reserve[]: 0 1880 258420 258420 <4>[530807.672457] Node 0 DMA32 free:387068kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[530807.715512] lowmem_reserve[]: 0 0 256540 256540 <4>[530807.721001] Node 0 Normal free:255788kB min:67088kB low:83860kB high:100632kB active_anon:338448kB inactive_anon:298888kB active_file:65514660kB inactive_file:65308140kB unevictable:39628kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:50112kB writeback:0kB mapped:45024kB shmem:547480kB slab_reclaimable:124241252kB slab_unreclaimable:4609836kB kernel_stack:74768kB pagetables:6116kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[530807.771220] lowmem_reserve[]: 0 0 0 0 <4>[530807.775725] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[530807.788346] Node 0 DMA32: 7*4kB 6*8kB 5*16kB 9*32kB 7*64kB 5*128kB 6*256kB 10*512kB 8*1024kB 5*2048kB 88*4096kB = 387068kB <4>[530807.801579] Node 0 Normal: 56764*4kB 2420*8kB 206*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 249712kB <4>[530807.815523] 32844748 total pagecache pages <4>[530807.820433] 0 pages in swap cache <4>[530807.824455] Swap cache stats: add 0, delete 0, find 0/0 <4>[530807.830614] Free swap = 0kB <4>[530807.834183] Total swap = 0kB <6>[530808.243137] 67108863 pages RAM <6>[530808.252120] 1015827 pages reserved <6>[530808.256239] 32339642 pages shared <6>[530808.260258] 33353100 pages non-shared <4>[530808.264731] dsm_sa_datamgrd: page allocation failure. order:1, mode:0x20 <4>[530808.272548] Pid: 11764, comm: dsm_sa_datamgrd Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[530808.282520] Call Trace: <4>[530808.285568] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[530808.293193] [] ? __alloc_pages_nodemask+0x7ff/0x950 <4>[530808.300817] [] ? alloc_pages_current+0xaa/0x110 <4>[530808.308046] [] ? __get_free_pages+0xe/0x50 <4>[530808.314818] [] ? swiotlb_alloc_coherent+0x5d/0x130 <4>[530808.322341] [] ? x86_swiotlb_alloc_coherent+0x61/0x70 <4>[530808.330167] [] ? pci_alloc_consistent+0x5a/0xc0 [mpt3sas] <4>[530808.338372] [] ? _ctl_do_mpt_command+0x284/0xcf0 [mpt3sas] <4>[530808.346895] [] ? _ctl_ioctl_main+0x58b/0x1120 [mpt3sas] <4>[530808.354930] [] ? __do_page_fault+0x1f4/0x500 <4>[530808.361869] [] ? _ctl_ioctl+0x16/0x20 [mpt3sas] <4>[530808.369119] [] ? vfs_ioctl+0x22/0xa0 <4>[530808.375300] [] ? do_nanosleep+0x93/0xc0 <4>[530808.381751] [] ? do_vfs_ioctl+0x84/0x580 <4>[530808.388302] [] ? hrtimer_wakeup+0x0/0x30 <4>[530808.394863] [] ? sys_ioctl+0x81/0xa0 <4>[530808.401027] [] ? system_call_fastpath+0x16/0x1b <6>[530808.408320] Mem-Info: <4>[530808.411188] Node 0 DMA per-cpu: <4>[530808.415030] CPU 0: hi: 0, btch: 1 usd: 0 <4>[530808.420747] CPU 1: hi: 0, btch: 1 usd: 0 <4>[530808.426417] CPU 2: hi: 0, btch: 1 usd: 0 <4>[530808.432192] CPU 3: hi: 0, btch: 1 usd: 0 <4>[530808.437878] CPU 4: hi: 0, btch: 1 usd: 0 <4>[530808.443553] CPU 5: hi: 0, btch: 1 usd: 0 <4>[530808.449226] CPU 6: hi: 0, btch: 1 usd: 0 <4>[530808.454903] CPU 7: hi: 0, btch: 1 usd: 0 <4>[530808.460686] Node 0 DMA32 per-cpu: <4>[530808.464763] CPU 0: hi: 186, btch: 31 usd: 0 <4>[530808.470436] CPU 1: hi: 186, btch: 31 usd: 0 <4>[530808.476108] CPU 2: hi: 186, btch: 31 usd: 0 <4>[530808.481780] CPU 3: hi: 186, btch: 31 usd: 0 <4>[530808.487462] CPU 4: hi: 186, btch: 31 usd: 0 <4>[530808.493146] CPU 5: hi: 186, btch: 31 usd: 0 <4>[530808.498849] CPU 6: hi: 186, btch: 31 usd: 0 <4>[530808.504543] CPU 7: hi: 186, btch: 31 usd: 0 <4>[530808.510227] Node 0 Normal per-cpu: <4>[530808.514368] CPU 0: hi: 186, btch: 31 usd: 214 <4>[530808.520071] CPU 1: hi: 186, btch: 31 usd: 223 <4>[530808.525758] CPU 2: hi: 186, btch: 31 usd: 198 <4>[530808.531430] CPU 3: hi: 186, btch: 31 usd: 174 <4>[530808.537102] CPU 4: hi: 186, btch: 31 usd: 200 <4>[530808.542776] CPU 5: hi: 186, btch: 31 usd: 182 <4>[530808.548452] CPU 6: hi: 186, btch: 31 usd: 51 <4>[530808.554128] CPU 7: hi: 186, btch: 31 usd: 161 <4>[530808.559803] active_anon:84612 inactive_anon:74722 isolated_anon:0 <4>[530808.559803] active_file:16375873 inactive_file:16329061 isolated_file:0 <4>[530808.559804] unevictable:9907 dirty:12528 writeback:0 unstable:0 <4>[530808.559804] free:174928 slab_reclaimable:31050765 slab_unreclaimable:1152407 <4>[530808.559804] mapped:11256 shmem:136870 pagetables:1529 bounce:0 <4>[530808.597227] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[530808.639178] lowmem_reserve[]: 0 1880 258420 258420 <4>[530808.644938] Node 0 DMA32 free:387068kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[530808.687963] lowmem_reserve[]: 0 0 256540 256540 <4>[530808.693408] Node 0 Normal free:289464kB min:67088kB low:83860kB high:100632kB active_anon:338448kB inactive_anon:298888kB active_file:65506984kB inactive_file:65317020kB unevictable:39628kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:50112kB writeback:0kB mapped:45024kB shmem:547480kB slab_reclaimable:124206164kB slab_unreclaimable:4609628kB kernel_stack:74768kB pagetables:6116kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[530808.743644] lowmem_reserve[]: 0 0 0 0 <4>[530808.748156] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[530808.760796] Node 0 DMA32: 7*4kB 6*8kB 5*16kB 9*32kB 7*64kB 5*128kB 6*256kB 10*512kB 8*1024kB 5*2048kB 88*4096kB = 387068kB <4>[530808.773897] Node 0 Normal: 67889*4kB 3119*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 296508kB <4>[530808.787597] 32841087 total pagecache pages <4>[530808.792502] 0 pages in swap cache <4>[530808.796531] Swap cache stats: add 0, delete 0, find 0/0 <4>[530808.802787] Free swap = 0kB <4>[530808.806331] Total swap = 0kB <6>[530809.220045] 67108863 pages RAM <6>[530809.223786] 1015827 pages reserved <6>[530809.228010] 32277655 pages shared <6>[530809.232043] 33285138 pages non-shared <4>[532334.292685] dsm_sa_datamgrd: page allocation failure. order:1, mode:0x8020 <4>[532334.300688] Pid: 11764, comm: dsm_sa_datamgrd Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[532334.310660] Call Trace: <4>[532334.313720] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[532334.321340] [] ? try_to_del_timer_sync+0x7b/0xe0 <4>[532334.328679] [] ? dma_generic_alloc_coherent+0xa6/0x160 <4>[532334.336590] [] ? x86_swiotlb_alloc_coherent+0x31/0x70 <4>[532334.344415] [] ? pci_alloc_consistent+0x5a/0xc0 [mpt3sas] <4>[532334.352620] [] ? _ctl_do_mpt_command+0x284/0xcf0 [mpt3sas] <4>[532334.361142] [] ? _ctl_ioctl_main+0x58b/0x1120 [mpt3sas] <4>[532334.369148] [] ? __do_page_fault+0x1f4/0x500 <4>[532334.376098] [] ? _ctl_ioctl+0x16/0x20 [mpt3sas] <4>[532334.383327] [] ? vfs_ioctl+0x22/0xa0 <4>[532334.389495] [] ? do_nanosleep+0x93/0xc0 <4>[532334.395946] [] ? do_vfs_ioctl+0x84/0x580 <4>[532334.402499] [] ? hrtimer_wakeup+0x0/0x30 <4>[532334.409053] [] ? sys_ioctl+0x81/0xa0 <4>[532334.415218] [] ? do_device_not_available+0xe/0x10 <4>[532334.422651] [] ? system_call_fastpath+0x16/0x1b <6>[532334.429885] Mem-Info: <4>[532334.432742] Node 0 DMA per-cpu: <4>[532334.436587] CPU 0: hi: 0, btch: 1 usd: 0 <4>[532334.442261] CPU 1: hi: 0, btch: 1 usd: 0 <4>[532334.447938] CPU 2: hi: 0, btch: 1 usd: 0 <4>[532334.453612] CPU 3: hi: 0, btch: 1 usd: 0 <4>[532334.459286] CPU 4: hi: 0, btch: 1 usd: 0 <4>[532334.464960] CPU 5: hi: 0, btch: 1 usd: 0 <4>[532334.470638] CPU 6: hi: 0, btch: 1 usd: 0 <4>[532334.476314] CPU 7: hi: 0, btch: 1 usd: 0 <4>[532334.481986] Node 0 DMA32 per-cpu: <4>[532334.486023] CPU 0: hi: 186, btch: 31 usd: 0 <4>[532334.491699] CPU 1: hi: 186, btch: 31 usd: 0 <4>[532334.497374] CPU 2: hi: 186, btch: 31 usd: 0 <4>[532334.503052] CPU 3: hi: 186, btch: 31 usd: 0 <4>[532334.508730] CPU 4: hi: 186, btch: 31 usd: 0 <4>[532334.514416] CPU 5: hi: 186, btch: 31 usd: 0 <4>[532334.520084] CPU 6: hi: 186, btch: 31 usd: 0 <4>[532334.525773] CPU 7: hi: 186, btch: 31 usd: 0 <4>[532334.531454] Node 0 Normal per-cpu: <4>[532334.535590] CPU 0: hi: 186, btch: 31 usd: 181 <4>[532334.541264] CPU 1: hi: 186, btch: 31 usd: 182 <4>[532334.546939] CPU 2: hi: 186, btch: 31 usd: 216 <4>[532334.552610] CPU 3: hi: 186, btch: 31 usd: 202 <4>[532334.558286] CPU 4: hi: 186, btch: 31 usd: 48 <4>[532334.563960] CPU 5: hi: 186, btch: 31 usd: 164 <4>[532334.569637] CPU 6: hi: 186, btch: 31 usd: 206 <4>[532334.575313] CPU 7: hi: 186, btch: 31 usd: 71 <4>[532334.580988] active_anon:93111 inactive_anon:66914 isolated_anon:0 <4>[532334.580989] active_file:15903503 inactive_file:15857071 isolated_file:0 <4>[532334.580989] unevictable:9907 dirty:8567 writeback:0 unstable:0 <4>[532334.580989] free:124844 slab_reclaimable:32079905 slab_unreclaimable:1116676 <4>[532334.580990] mapped:11216 shmem:137241 pagetables:1600 bounce:0 <4>[532334.618333] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[532334.660296] lowmem_reserve[]: 0 1880 258420 258420 <4>[532334.666052] Node 0 DMA32 free:387068kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[532334.709143] lowmem_reserve[]: 0 0 256540 256540 <4>[532334.714598] Node 0 Normal free:119384kB min:67088kB low:83860kB high:100632kB active_anon:372444kB inactive_anon:267656kB active_file:63604592kB inactive_file:63429448kB unevictable:39628kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:34268kB writeback:0kB mapped:44864kB shmem:548964kB slab_reclaimable:128305652kB slab_unreclaimable:4466704kB kernel_stack:74816kB pagetables:6400kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[532334.764806] lowmem_reserve[]: 0 0 0 0 <4>[532334.769294] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[532334.781919] Node 0 DMA32: 7*4kB 6*8kB 5*16kB 9*32kB 7*64kB 5*128kB 6*256kB 10*512kB 8*1024kB 5*2048kB 88*4096kB = 387068kB <4>[532334.794991] Node 0 Normal: 23425*4kB 904*8kB 231*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 104628kB <4>[532334.808791] 31898589 total pagecache pages <4>[532334.818981] 0 pages in swap cache <4>[532334.823035] Swap cache stats: add 0, delete 0, find 0/0 <4>[532334.829224] Free swap = 0kB <4>[532334.832770] Total swap = 0kB <6>[532335.238144] 67108863 pages RAM <6>[532335.241879] 1015827 pages reserved <6>[532335.245999] 31235554 pages shared <6>[532335.250013] 34493501 pages non-shared <4>[532735.708335] swapper: page allocation failure. order:1, mode:0x20 <4>[532735.715394] Pid: 0, comm: swapper Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[532735.724196] Call Trace: <4>[532735.727255] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[532735.735788] [] ? kmem_getpages+0x62/0x170 <4>[532735.742427] [] ? fallback_alloc+0x1ba/0x270 <4>[532735.749288] [] ? cache_grow+0x2cf/0x320 <4>[532735.755751] [] ? ____cache_alloc_node+0x99/0x160 <4>[532735.763076] [] ? kmem_cache_alloc_node_trace+0x90/0x200 <4>[532735.771084] [] ? __kmalloc_node+0x4d/0x60 <4>[532735.777730] [] ? __alloc_skb+0x7a/0x190 <4>[532735.784179] [] ? dev_alloc_skb+0x1d/0x40 <4>[532735.790736] [] ? ipoib_alloc_rx_skb+0x3f/0x200 [ib_ipoib] <4>[532735.798943] [] ? mlx4_ib_poll_cq+0x52a/0xd30 [mlx4_ib] <4>[532735.806856] [] ? ipoib_ib_handle_rx_wc+0x8c/0x300 [ib_ipoib] <4>[532735.815574] [] ? ipoib_poll+0x14b/0x180 [ib_ipoib] <4>[532735.823098] [] ? net_rx_action+0x103/0x300 <4>[532735.829840] [] ? __do_softirq+0xe5/0x230 <4>[532735.836397] [] ? mlx4_msi_x_interrupt+0x14/0x20 [mlx4_core] <4>[532735.845004] [] ? call_softirq+0x1c/0x30 <4>[532735.851478] [] ? do_softirq+0x65/0xa0 <4>[532735.857729] [] ? irq_exit+0x85/0x90 <4>[532735.863790] [] ? do_IRQ+0x75/0xf0 <4>[532735.869656] [] ? ret_from_intr+0x0/0x11 <4>[532735.876107] [] ? intel_idle+0xfe/0x1b0 <4>[532735.883177] [] ? intel_idle+0xe1/0x1b0 <4>[532735.889543] [] ? sched_clock+0x9/0x10 <4>[532735.895812] [] ? sched_clock_cpu+0xcd/0x110 <4>[532735.902663] [] ? cpuidle_idle_call+0x7a/0xe0 <4>[532735.909604] [] ? cpu_idle+0xb6/0x110 <4>[532735.915785] [] ? start_secondary+0x2c0/0x316 <4>[533555.297831] dsm_sa_datamgrd: page allocation failure. order:1, mode:0x8020 <4>[533555.305845] Pid: 11764, comm: dsm_sa_datamgrd Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[533555.315823] Call Trace: <4>[533555.318884] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[533555.326512] [] ? try_to_del_timer_sync+0x7b/0xe0 <4>[533555.333847] [] ? dma_generic_alloc_coherent+0xa6/0x160 <4>[533555.341757] [] ? x86_swiotlb_alloc_coherent+0x31/0x70 <4>[533555.349588] [] ? pci_alloc_consistent+0x5a/0xc0 [mpt3sas] <4>[533555.357801] [] ? _ctl_do_mpt_command+0x284/0xcf0 [mpt3sas] <4>[533555.366329] [] ? _ctl_ioctl_main+0x58b/0x1120 [mpt3sas] <4>[533555.374343] [] ? __do_page_fault+0x1f4/0x500 <4>[533555.381299] [] ? _ctl_ioctl+0x16/0x20 [mpt3sas] <4>[533555.388553] [] ? vfs_ioctl+0x22/0xa0 <4>[533555.394724] [] ? do_nanosleep+0x93/0xc0 <4>[533555.401188] [] ? do_vfs_ioctl+0x84/0x580 <4>[533555.407741] [] ? hrtimer_wakeup+0x0/0x30 <4>[533555.414292] [] ? sys_ioctl+0x81/0xa0 <4>[533555.420455] [] ? do_device_not_available+0xe/0x10 <4>[533555.427917] [] ? system_call_fastpath+0x16/0x1b <6>[533555.435150] Mem-Info: <4>[533555.438008] Node 0 DMA per-cpu: <4>[533555.441898] CPU 0: hi: 0, btch: 1 usd: 0 <4>[533555.447577] CPU 1: hi: 0, btch: 1 usd: 0 <4>[533555.453248] CPU 2: hi: 0, btch: 1 usd: 0 <4>[533555.458927] CPU 3: hi: 0, btch: 1 usd: 0 <4>[533555.464612] CPU 4: hi: 0, btch: 1 usd: 0 <4>[533555.470291] CPU 5: hi: 0, btch: 1 usd: 0 <4>[533555.475967] CPU 6: hi: 0, btch: 1 usd: 0 <4>[533555.481644] CPU 7: hi: 0, btch: 1 usd: 0 <4>[533555.487326] Node 0 DMA32 per-cpu: <4>[533555.491397] CPU 0: hi: 186, btch: 31 usd: 0 <4>[533555.497095] CPU 1: hi: 186, btch: 31 usd: 0 <4>[533555.502771] CPU 2: hi: 186, btch: 31 usd: 0 <4>[533555.508445] CPU 3: hi: 186, btch: 31 usd: 0 <4>[533555.514143] CPU 4: hi: 186, btch: 31 usd: 0 <4>[533555.519823] CPU 5: hi: 186, btch: 31 usd: 0 <4>[533555.525510] CPU 6: hi: 186, btch: 31 usd: 0 <4>[533555.531178] CPU 7: hi: 186, btch: 31 usd: 0 <4>[533555.536852] Node 0 Normal per-cpu: <4>[533555.541013] CPU 0: hi: 186, btch: 31 usd: 202 <4>[533555.546716] CPU 1: hi: 186, btch: 31 usd: 198 <4>[533555.552403] CPU 2: hi: 186, btch: 31 usd: 162 <4>[533555.558084] CPU 3: hi: 186, btch: 31 usd: 102 <4>[533555.563762] CPU 4: hi: 186, btch: 31 usd: 35 <4>[533555.569437] CPU 5: hi: 186, btch: 31 usd: 170 <4>[533555.575142] CPU 6: hi: 186, btch: 31 usd: 181 <4>[533555.580827] CPU 7: hi: 186, btch: 31 usd: 175 <4>[533555.586527] active_anon:92947 inactive_anon:66976 isolated_anon:0 <4>[533555.586527] active_file:15519103 inactive_file:15516791 isolated_file:0 <4>[533555.586528] unevictable:9907 dirty:8257 writeback:0 unstable:0 <4>[533555.586528] free:169263 slab_reclaimable:32778572 slab_unreclaimable:1098697 <4>[533555.586529] mapped:11190 shmem:137130 pagetables:1601 bounce:0 <4>[533555.623850] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[533555.665839] lowmem_reserve[]: 0 1880 258420 258420 <4>[533555.671597] Node 0 DMA32 free:387068kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[533555.714632] lowmem_reserve[]: 0 0 256540 256540 <4>[533555.720103] Node 0 Normal free:252916kB min:67088kB low:83860kB high:100632kB active_anon:371788kB inactive_anon:267904kB active_file:62082620kB inactive_file:62068716kB unevictable:39628kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:33028kB writeback:0kB mapped:44760kB shmem:548520kB slab_reclaimable:131128644kB slab_unreclaimable:4394788kB kernel_stack:74816kB pagetables:6404kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[533555.770244] lowmem_reserve[]: 0 0 0 0 <4>[533555.774784] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[533555.787409] Node 0 DMA32: 7*4kB 6*8kB 5*16kB 9*32kB 7*64kB 5*128kB 6*256kB 10*512kB 8*1024kB 5*2048kB 88*4096kB = 387068kB <4>[533555.800500] Node 0 Normal: 59148*4kB 382*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 239648kB <4>[533555.814124] 31177699 total pagecache pages <4>[533555.819024] 0 pages in swap cache <4>[533555.823049] Swap cache stats: add 0, delete 0, find 0/0 <4>[533555.829205] Free swap = 0kB <4>[533555.832740] Total swap = 0kB <6>[533556.224698] 67108863 pages RAM <6>[533556.228433] 1015827 pages reserved <6>[533556.232544] 30267755 pages shared <6>[533556.236570] 35433591 pages non-shared <4>[533556.240986] dsm_sa_datamgrd: page allocation failure. order:1, mode:0x20 <4>[533556.248804] Pid: 11764, comm: dsm_sa_datamgrd Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[533556.258780] Call Trace: <4>[533556.261840] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[533556.269466] [] ? __alloc_pages_nodemask+0x7ff/0x950 <4>[533556.277101] [] ? alloc_pages_current+0xaa/0x110 <4>[533556.284347] [] ? __get_free_pages+0xe/0x50 <4>[533556.291096] [] ? swiotlb_alloc_coherent+0x5d/0x130 <4>[533556.298630] [] ? x86_swiotlb_alloc_coherent+0x61/0x70 <4>[533556.306468] [] ? pci_alloc_consistent+0x5a/0xc0 [mpt3sas] <4>[533556.314687] [] ? _ctl_do_mpt_command+0x284/0xcf0 [mpt3sas] <4>[533556.323205] [] ? _ctl_ioctl_main+0x58b/0x1120 [mpt3sas] <4>[533556.331217] [] ? __do_page_fault+0x1f4/0x500 <4>[533556.338179] [] ? _ctl_ioctl+0x16/0x20 [mpt3sas] <4>[533556.345411] [] ? vfs_ioctl+0x22/0xa0 <4>[533556.351574] [] ? do_nanosleep+0x93/0xc0 <4>[533556.358028] [] ? do_vfs_ioctl+0x84/0x580 <4>[533556.364582] [] ? hrtimer_wakeup+0x0/0x30 <4>[533556.371143] [] ? sys_ioctl+0x81/0xa0 <4>[533556.377302] [] ? do_device_not_available+0xe/0x10 <4>[533556.384730] [] ? system_call_fastpath+0x16/0x1b <6>[533556.392054] Mem-Info: <4>[533556.394926] Node 0 DMA per-cpu: <4>[533556.398776] CPU 0: hi: 0, btch: 1 usd: 0 <4>[533556.404451] CPU 1: hi: 0, btch: 1 usd: 0 <4>[533556.410129] CPU 2: hi: 0, btch: 1 usd: 0 <4>[533556.415809] CPU 3: hi: 0, btch: 1 usd: 0 <4>[533556.421485] CPU 4: hi: 0, btch: 1 usd: 0 <4>[533556.427166] CPU 5: hi: 0, btch: 1 usd: 0 <4>[533556.432842] CPU 6: hi: 0, btch: 1 usd: 0 <4>[533556.438529] CPU 7: hi: 0, btch: 1 usd: 0 <4>[533556.444212] Node 0 DMA32 per-cpu: <4>[533556.448256] CPU 0: hi: 186, btch: 31 usd: 0 <4>[533556.453932] CPU 1: hi: 186, btch: 31 usd: 0 <4>[533556.459609] CPU 2: hi: 186, btch: 31 usd: 0 <4>[533556.465284] CPU 3: hi: 186, btch: 31 usd: 0 <4>[533556.470963] CPU 4: hi: 186, btch: 31 usd: 0 <4>[533556.476634] CPU 5: hi: 186, btch: 31 usd: 0 <4>[533556.482322] CPU 6: hi: 186, btch: 31 usd: 0 <4>[533556.488003] CPU 7: hi: 186, btch: 31 usd: 0 <4>[533556.493679] Node 0 Normal per-cpu: <4>[533556.497819] CPU 0: hi: 186, btch: 31 usd: 194 <4>[533556.503494] CPU 1: hi: 186, btch: 31 usd: 198 <4>[533556.514415] CPU 2: hi: 186, btch: 31 usd: 162 <4>[533556.520103] CPU 3: hi: 186, btch: 31 usd: 231 <4>[533556.525780] CPU 4: hi: 186, btch: 31 usd: 207 <4>[533556.531460] CPU 5: hi: 186, btch: 31 usd: 218 <4>[533556.537145] CPU 6: hi: 186, btch: 31 usd: 153 <4>[533556.542819] CPU 7: hi: 186, btch: 31 usd: 236 <4>[533556.548508] active_anon:92947 inactive_anon:66976 isolated_anon:0 <4>[533556.548508] active_file:15520491 inactive_file:15524349 isolated_file:0 <4>[533556.548509] unevictable:9907 dirty:8257 writeback:0 unstable:0 <4>[533556.548509] free:158848 slab_reclaimable:32780188 slab_unreclaimable:1098553 <4>[533556.548510] mapped:11190 shmem:137130 pagetables:1601 bounce:0 <4>[533556.585930] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[533556.627895] lowmem_reserve[]: 0 1880 258420 258420 <4>[533556.633655] Node 0 DMA32 free:387068kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[533556.676716] lowmem_reserve[]: 0 0 256540 256540 <4>[533556.682178] Node 0 Normal free:214232kB min:67088kB low:83860kB high:100632kB active_anon:371788kB inactive_anon:267904kB active_file:62084292kB inactive_file:62106320kB unevictable:39628kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:33028kB writeback:0kB mapped:44760kB shmem:548520kB slab_reclaimable:131126184kB slab_unreclaimable:4394212kB kernel_stack:74816kB pagetables:6404kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[533556.732334] lowmem_reserve[]: 0 0 0 0 <4>[533556.736819] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[533556.749438] Node 0 DMA32: 7*4kB 6*8kB 5*16kB 9*32kB 7*64kB 5*128kB 6*256kB 10*512kB 8*1024kB 5*2048kB 88*4096kB = 387068kB <4>[533556.762536] Node 0 Normal: 46907*4kB 1364*8kB 231*16kB 30*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 203196kB <4>[533556.776497] 31188507 total pagecache pages <4>[533556.781395] 0 pages in swap cache <4>[533556.785417] Swap cache stats: add 0, delete 0, find 0/0 <4>[533556.791579] Free swap = 0kB <4>[533556.795134] Total swap = 0kB <6>[533557.191010] 67108863 pages RAM <6>[533557.194745] 1015827 pages reserved <6>[533557.198869] 30278668 pages shared <6>[533557.202894] 35430402 pages non-shared <4>[533651.354753] swapper: page allocation failure. order:1, mode:0x20 <4>[533651.361788] Pid: 0, comm: swapper Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[533651.370591] Call Trace: <4>[533651.373641] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[533651.382247] [] ? check_preempt_curr+0x20/0x90 <4>[533651.389280] [] ? kmem_getpages+0x62/0x170 <4>[533651.395924] [] ? fallback_alloc+0x1ba/0x270 <4>[533651.402783] [] ? cache_grow+0x2cf/0x320 <4>[533651.409233] [] ? ____cache_alloc_node+0x99/0x160 <4>[533651.416564] [] ? kmem_cache_alloc_node_trace+0x90/0x200 <4>[533651.424571] [] ? __kmalloc_node+0x4d/0x60 <4>[533651.431216] [] ? __alloc_skb+0x7a/0x190 <4>[533651.437676] [] ? dev_alloc_skb+0x1d/0x40 <4>[533651.444233] [] ? ipoib_alloc_rx_skb+0x3f/0x200 [ib_ipoib] <4>[533651.452432] [] ? mlx4_ib_poll_cq+0x52a/0xd30 [mlx4_ib] <4>[533651.460343] [] ? ipoib_ib_handle_rx_wc+0x8c/0x300 [ib_ipoib] <4>[533651.469071] [] ? ipoib_poll+0x14b/0x180 [ib_ipoib] <4>[533651.476592] [] ? net_rx_action+0x103/0x300 <4>[533651.483357] [] ? __do_softirq+0xe5/0x230 <4>[533651.489917] [] ? mlx4_msi_x_interrupt+0x14/0x20 [mlx4_core] <4>[533651.498558] [] ? call_softirq+0x1c/0x30 <4>[533651.505010] [] ? do_softirq+0x65/0xa0 <4>[533651.511267] [] ? irq_exit+0x85/0x90 <4>[533651.517332] [] ? do_IRQ+0x75/0xf0 <4>[533651.523200] [] ? ret_from_intr+0x0/0x11 <4>[533651.529656] [] ? intel_idle+0xfe/0x1b0 <4>[533651.536729] [] ? intel_idle+0xe1/0x1b0 <4>[533651.543088] [] ? sched_clock+0x9/0x10 <4>[533651.549448] [] ? sched_clock_cpu+0xcd/0x110 <4>[533651.556293] [] ? cpuidle_idle_call+0x7a/0xe0 <4>[533651.563233] [] ? cpu_idle+0xb6/0x110 <4>[533651.569394] [] ? start_secondary+0x2c0/0x316 <4>[533651.576354] swapper: page allocation failure. order:1, mode:0x20 <4>[533651.583388] Pid: 0, comm: swapper Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[533651.592196] Call Trace: <4>[533651.595249] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[533651.603786] [] ? scheduler_tick+0x11e/0x260 <4>[533651.610647] [] ? mlx4_msi_x_interrupt+0x14/0x20 [mlx4_core] <4>[533651.619260] [] ? kmem_getpages+0x62/0x170 <4>[533651.625910] [] ? fallback_alloc+0x1ba/0x270 <4>[533651.632760] [] ? cache_grow+0x2cf/0x320 <4>[533651.639215] [] ? ____cache_alloc_node+0x99/0x160 <4>[533651.646542] [] ? kmem_cache_alloc_node_trace+0x90/0x200 <4>[533651.654550] [] ? __kmalloc_node+0x4d/0x60 <4>[533651.661197] [] ? __alloc_skb+0x7a/0x190 <4>[533651.667648] [] ? dev_alloc_skb+0x1d/0x40 <4>[533651.674212] [] ? ipoib_alloc_rx_skb+0x3f/0x200 [ib_ipoib] <4>[533651.682412] [] ? ipoib_ib_post_receive+0x7e/0x100 [ib_ipoib] <4>[533651.691131] [] ? ipoib_ib_handle_rx_wc+0x8c/0x300 [ib_ipoib] <4>[533651.699870] [] ? ipoib_poll+0x14b/0x180 [ib_ipoib] <4>[533651.707388] [] ? net_rx_action+0x103/0x300 <4>[533651.714135] [] ? __do_softirq+0xe5/0x230 <4>[533651.720710] [] ? mlx4_msi_x_interrupt+0x14/0x20 [mlx4_core] <4>[533651.729331] [] ? call_softirq+0x1c/0x30 <4>[533651.735791] [] ? do_softirq+0x65/0xa0 <4>[533651.742061] [] ? irq_exit+0x85/0x90 <4>[533651.748130] [] ? do_IRQ+0x75/0xf0 <4>[533651.753997] [] ? ret_from_intr+0x0/0x11 <4>[533651.760468] [] ? intel_idle+0xfe/0x1b0 <4>[533651.767527] [] ? intel_idle+0xe1/0x1b0 <4>[533651.773882] [] ? sched_clock+0x9/0x10 <4>[533651.780145] [] ? sched_clock_cpu+0xcd/0x110 <4>[533651.786986] [] ? cpuidle_idle_call+0x7a/0xe0 <4>[533651.793924] [] ? cpu_idle+0xb6/0x110 <4>[533651.800085] [] ? start_secondary+0x2c0/0x316 <4>[543680.048794] Lustre: atlas1-MDT0000: Client 79af5404-fb8f-24b7-458b-11428b4737bb (at 16210@gni100) reconnecting <6>[543680.060543] Lustre: atlas1-MDT0000: Connection restored to 79af5404-fb8f-24b7-458b-11428b4737bb (at 16210@gni100) <6>[543680.072607] Lustre: Skipped 1 previous similar message <4>[552771.747483] Lustre: atlas1-MDT0000: haven't heard from client df34b9ce-5df1-67d7-b884-ff134c8609df (at 7208@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0e395000, cur 1487047873 expire 1487046973 last 1487046521 <4>[552771.772757] Lustre: Skipped 167 previous similar messages <4>[557955.738626] Lustre: atlas1-MDT0000: haven't heard from client 381d1b85-e60d-0315-4d77-2d0081b382b7 (at 13577@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880321dea000, cur 1487053057 expire 1487052157 last 1487051705 <4>[557955.763909] Lustre: Skipped 1 previous similar message <4>[567562.728760] Lustre: atlas1-MDT0000: haven't heard from client d1caf2a7-b948-4662-a3f6-c4b9ccee69a6 (at 18534@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883bd6f7e400, cur 1487062664 expire 1487061764 last 1487061312 <6>[580316.690551] Lustre: atlas1-MDT0000: Connection restored to eeb5827c-a783-8b42-5e11-740454a0dc79 (at 0@lo) <6>[580316.701911] Lustre: client wants to enable acl, but mdt not! <4>[580316.890974] Lustre: Mounted atlas1-client <4>[582825.554483] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[582828.780295] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[582831.249429] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[586086.345432] Lustre: Unmounted atlas1-client <6>[586868.114972] Lustre: atlas1-MDT0000: Connection restored to 8fb8b502-362f-7021-86d8-230b4bffed18 (at 10.39.232.64@o2ib6) <6>[586869.513492] Lustre: atlas1-MDT0000: Connection restored to 13ce34fc-050c-2e38-b99b-190b0b640819 (at 10.39.232.63@o2ib6) <6>[586871.072026] Lustre: atlas1-MDT0000: Connection restored to 0fd268b6-b6bb-cc78-ee5d-6381ff7023a0 (at 10.39.232.87@o2ib6) <6>[586871.084657] Lustre: Skipped 1 previous similar message <6>[586874.594252] Lustre: atlas1-MDT0000: Connection restored to 2a5e9392-78c2-f9de-a5f4-ee0278da5be5 (at 10.39.232.95@o2ib6) <6>[586874.606850] Lustre: Skipped 2 previous similar messages <6>[586879.030167] Lustre: atlas1-MDT0000: Connection restored to 36a38097-23ee-0940-82bd-8b9977dc5b4c (at 10.39.232.90@o2ib6) <6>[586879.042792] Lustre: Skipped 2 previous similar messages <6>[586900.331347] Lustre: atlas1-MDT0000: Connection restored to 668f9422-269e-ff31-ac9a-79da33a9fe59 (at 10.39.232.93@o2ib6) <6>[587459.663766] Lustre: atlas1-MDT0000: Connection restored to d66a314f-34c4-2442-29ac-667012aa506c (at 10.39.232.73@o2ib6) <4>[587677.697827] Lustre: atlas1-MDT0000: haven't heard from client 71430c47-26e5-0d5b-e10c-5aa4661ad234 (at 16955@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880eb93dd400, cur 1487082779 expire 1487081879 last 1487081427 <4>[588128.698313] Lustre: atlas1-MDT0000: haven't heard from client d66a314f-34c4-2442-29ac-667012aa506c (at 10.39.232.73@o2ib6) in 951 seconds. I think it's dead, and I am evicting it. exp ffff881e474ea800, cur 1487083230 expire 1487082330 last 1487082279 <4>[588128.724079] Lustre: Skipped 222 previous similar messages <3>[591974.247548] LustreError: 15968:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20039ef32:0x1a2a8:0x0]: rc = -2 <3>[591974.262673] LustreError: 15968:0:(mdt_handler.c:893:mdt_getattr_internal()) Skipped 3 previous similar messages <3>[592275.487734] LustreError: 16368:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a06a7:0x9:0x0]: rc = -2 <4>[597220.149201] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[597223.103209] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[597225.133870] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[597902.224714] ib_cm/7: page allocation failure. order:5, mode:0xd0 <4>[597902.231757] Pid: 2773, comm: ib_cm/7 Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[597902.240860] Call Trace: <4>[597902.243920] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[597902.251546] [] ? alloc_vmap_area+0x27a/0x390 <4>[597902.258495] [] ? kmem_getpages+0x62/0x170 <4>[597902.265155] [] ? fallback_alloc+0x1ba/0x270 <4>[597902.272006] [] ? cache_grow+0x2cf/0x320 <4>[597902.278471] [] ? ____cache_alloc_node+0x99/0x160 <4>[597902.285816] [] ? create_qp_common+0xedb/0xf50 [mlx4_ib] <4>[597902.293836] [] ? __kmalloc+0x199/0x230 <4>[597902.300202] [] ? create_qp_common+0xedb/0xf50 [mlx4_ib] <4>[597902.308219] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[597902.315840] [] ? mlx4_ib_create_qp+0x166/0x2b0 [mlx4_ib] <4>[597902.323962] [] ? ib_create_qp+0x41/0x1c0 [ib_core] <4>[597902.331495] [] ? rdma_create_qp+0x48/0xc0 [rdma_cm] <4>[597902.339127] [] ? kiblnd_create_conn+0xa1d/0x16e0 [ko2iblnd] <4>[597902.347757] [] ? kiblnd_passive_connect+0x84b/0x17b0 [ko2iblnd] <4>[597902.356766] [] ? fib_del_ifaddr+0x3a2/0x450 <4>[597902.363627] [] ? ib_find_cached_gid+0xec/0x110 [ib_core] <4>[597902.371742] [] ? kiblnd_cm_callback+0x6dd/0x2090 [ko2iblnd] <4>[597902.380371] [] ? cma_req_handler+0x371/0x640 [rdma_cm] <4>[597902.388308] [] ? rdma_port_get_link_layer+0x1b/0x60 [ib_core] <4>[597902.397139] [] ? cm_process_work+0x27/0x110 [ib_cm] <4>[597902.404771] [] ? cm_req_handler+0x6b5/0xac0 [ib_cm] <4>[597902.412405] [] ? cm_work_handler+0x0/0x1206 [ib_cm] <4>[597902.420038] [] ? cm_work_handler+0x0/0x1206 [ib_cm] <4>[597902.427668] [] ? cm_work_handler+0x135/0x1206 [ib_cm] <4>[597902.435493] [] ? prepare_to_wait+0x4e/0x80 <4>[597902.442261] [] ? cm_work_handler+0x0/0x1206 [ib_cm] <4>[597902.449907] [] ? worker_thread+0x170/0x2a0 <4>[597902.456662] [] ? autoremove_wake_function+0x0/0x40 <4>[597902.464197] [] ? worker_thread+0x0/0x2a0 <4>[597902.470763] [] ? kthread+0x9e/0xc0 <4>[597902.476731] [] ? child_rip+0xa/0x20 <4>[597902.482799] [] ? kthread+0x0/0xc0 <4>[597902.488689] [] ? child_rip+0x0/0x20 <6>[597902.494761] Mem-Info: <4>[597902.497628] Node 0 DMA per-cpu: <4>[597902.501476] CPU 0: hi: 0, btch: 1 usd: 0 <4>[597902.507182] CPU 1: hi: 0, btch: 1 usd: 0 <4>[597902.512862] CPU 2: hi: 0, btch: 1 usd: 0 <4>[597902.518550] CPU 3: hi: 0, btch: 1 usd: 0 <4>[597902.524230] CPU 4: hi: 0, btch: 1 usd: 0 <4>[597902.529913] CPU 5: hi: 0, btch: 1 usd: 0 <4>[597902.535595] CPU 6: hi: 0, btch: 1 usd: 0 <4>[597902.541277] CPU 7: hi: 0, btch: 1 usd: 0 <4>[597902.546965] Node 0 DMA32 per-cpu: <4>[597902.551014] CPU 0: hi: 186, btch: 31 usd: 0 <4>[597902.556699] CPU 1: hi: 186, btch: 31 usd: 0 <4>[597902.562387] CPU 2: hi: 186, btch: 31 usd: 0 <4>[597902.568066] CPU 3: hi: 186, btch: 31 usd: 0 <4>[597902.573746] CPU 4: hi: 186, btch: 31 usd: 0 <4>[597902.579431] CPU 5: hi: 186, btch: 31 usd: 0 <4>[597902.585127] CPU 6: hi: 186, btch: 31 usd: 0 <4>[597902.590801] CPU 7: hi: 186, btch: 31 usd: 0 <4>[597902.596492] Node 0 Normal per-cpu: <4>[597902.600643] CPU 0: hi: 186, btch: 31 usd: 131 <4>[597902.606324] CPU 1: hi: 186, btch: 31 usd: 159 <4>[597902.611995] CPU 2: hi: 186, btch: 31 usd: 180 <4>[597902.617675] CPU 3: hi: 186, btch: 31 usd: 174 <4>[597902.623352] CPU 4: hi: 186, btch: 31 usd: 27 <4>[597902.629030] CPU 5: hi: 186, btch: 31 usd: 50 <4>[597902.634712] CPU 6: hi: 186, btch: 31 usd: 171 <4>[597902.640391] CPU 7: hi: 186, btch: 31 usd: 0 <4>[597902.646075] active_anon:84499 inactive_anon:78248 isolated_anon:0 <4>[597902.646075] active_file:24108511 inactive_file:7642245 isolated_file:23 <4>[597902.646076] unevictable:10493 dirty:9552 writeback:0 unstable:0 <4>[597902.646076] free:7388568 slab_reclaimable:17114163 slab_unreclaimable:8826508 <4>[597902.646077] mapped:11585 shmem:140577 pagetables:1466 bounce:0 <4>[597902.683654] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[597902.725665] lowmem_reserve[]: 0 1880 258420 258420 <4>[597902.731437] Node 0 DMA32 free:387068kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[597902.774479] lowmem_reserve[]: 0 0 256540 256540 <4>[597902.779944] Node 0 Normal free:29219416kB min:67088kB low:83860kB high:100632kB active_anon:337996kB inactive_anon:312992kB active_file:96221872kB inactive_file:30712600kB unevictable:41972kB isolated(anon):0kB isolated(file):92kB present:262696960kB mlocked:0kB dirty:38208kB writeback:0kB mapped:46340kB shmem:562308kB slab_reclaimable:68456652kB slab_unreclaimable:35306032kB kernel_stack:74784kB pagetables:5864kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[597902.830390] lowmem_reserve[]: 0 0 0 0 <4>[597902.834889] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[597902.847512] Node 0 DMA32: 7*4kB 6*8kB 5*16kB 9*32kB 7*64kB 5*128kB 6*256kB 10*512kB 8*1024kB 5*2048kB 88*4096kB = 387068kB <4>[597902.860633] Node 0 Normal: 33793*4kB 758866*8kB 844927*16kB 249001*32kB 24466*64kB 6*128kB 6*256kB 3*512kB 2*1024kB 0*2048kB 0*4096kB = 29264676kB <4>[597902.876086] 31862482 total pagecache pages <4>[597902.880995] 0 pages in swap cache <4>[597902.885020] Swap cache stats: add 0, delete 0, find 0/0 <4>[597902.891186] Free swap = 0kB <4>[597902.894726] Total swap = 0kB <6>[597903.289520] 67108863 pages RAM <6>[597903.293257] 1015827 pages reserved <6>[597903.297377] 28940985 pages shared <6>[597903.301423] 29464605 pages non-shared <6>[597903.308430] Lustre: atlas1-MDT0000: Connection restored to 1be2489b-e6f0-3b9b-368d-3c5557da4a46 (at 10.36.202.44@o2ib) <3>[598045.382907] LustreError: 15664:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a0771:0x2343:0x0]: rc = -2 <4>[598890.681906] Lustre: atlas1-MDT0000: haven't heard from client f34ee6f1-4df9-9401-dcdb-204bd799422b (at 10.36.202.44@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883ecfd70800, cur 1487093992 expire 1487093092 last 1487092640 <3>[599223.998913] LustreError: 15789:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20039f830:0x170e2:0x0]: rc = -2 <3>[599275.537243] LustreError: 15773:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20039f830:0x176d5:0x0]: rc = -2 <4>[599808.680517] Lustre: atlas1-MDT0000: haven't heard from client a67e9494-3287-0cb1-8ac0-e6cb8aa75b16 (at 8095@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f1026cc00, cur 1487094910 expire 1487094010 last 1487093558 <3>[599991.006479] LustreError: 15934:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20039f830:0x19586:0x0]: rc = -2 <6>[600833.940676] Lustre: atlas1-MDT0000: Connection restored to 1007f51a-2a02-4254-8011-862619edd7c3 (at 40@gni100) <6>[600834.443055] Lustre: atlas1-MDT0000: Connection restored to f50e2e89-6904-0991-015b-f440efb6037d (at 1582@gni100) <6>[600834.454975] Lustre: Skipped 13 previous similar messages <6>[600835.617353] Lustre: atlas1-MDT0000: Connection restored to 25c61e9e-f2ea-fb15-f6fe-fb0a0479ddd4 (at 16936@gni100) <6>[600835.629375] Lustre: Skipped 13 previous similar messages <6>[600837.976961] Lustre: atlas1-MDT0000: Connection restored to 2e3fbc83-b504-d059-d826-b9aec8121b11 (at 38@gni100) <6>[600837.988690] Lustre: Skipped 11 previous similar messages <6>[600842.257121] Lustre: atlas1-MDT0000: Connection restored to c40e9715-9b99-962a-a70e-d3489de5f01b (at 47@gni100) <6>[600842.268882] Lustre: Skipped 45 previous similar messages <6>[600850.259129] Lustre: atlas1-MDT0000: Connection restored to b63f2383-7677-577e-a935-f91eb48309cf (at 1587@gni100) <6>[600850.271050] Lustre: Skipped 83 previous similar messages <6>[601367.829444] Lustre: atlas1-MDT0000: Connection restored to 1eff6663-cfa9-f3b8-24b5-e03c7768a7c7 (at 18388@gni100) <6>[601367.841462] Lustre: Skipped 49 previous similar messages <4>[602225.677061] Lustre: atlas1-MDT0000: haven't heard from client 3a073b68-600b-0b73-4600-926fdd079254 (at 3018@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8810c4798800, cur 1487097327 expire 1487096427 last 1487095975 <4>[602676.676537] Lustre: atlas1-MDT0000: haven't heard from client 069af209-6545-c0ac-0156-458f423325a5 (at 3018@gni100) in 1271 seconds. I think it's dead, and I am evicting it. exp ffff880909b4d400, cur 1487097778 expire 1487096878 last 1487096507 <4>[602676.701779] Lustre: Skipped 1 previous similar message <4>[603469.012364] Lustre: 15679:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1487098003/real 1487098003] req@ffff8810769f5380 x1558707304651264/t0(0) o104->atlas1-MDT0000@1494@gni100:15/16 lens 296/224 e 0 to 1 dl 1487098570 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[603469.043665] Lustre: 15679:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 1 previous similar message <4>[603496.961330] Lustre: 15546:0:(service.c:1336:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[603496.961331] req@ffff8828331f89c0 x1558697303869196/t0(0) o36->dd20153a-8521-3665-c176-6fbe4f953815@9228@gni100:18/0 lens 616/3128 e 12 to 0 dl 1487098603 ref 2 fl Interpret:/0/0 rc 0/0 <4>[603496.994329] Lustre: 15546:0:(service.c:1336:ptlrpc_at_send_early_reply()) Skipped 16 previous similar messages <4>[603752.007267] Lustre: atlas1-MDT0000: Client dd20153a-8521-3665-c176-6fbe4f953815 (at 9228@gni100) reconnecting <6>[603752.024057] Lustre: atlas1-MDT0000: Connection restored to dd20153a-8521-3665-c176-6fbe4f953815 (at 9228@gni100) <6>[603752.035980] Lustre: Skipped 80 previous similar messages <4>[603952.674849] Lustre: atlas1-MDT0000: haven't heard from client f3ce214c-dae4-320b-2a2d-070ce3975835 (at 1494@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880eada3f000, cur 1487099054 expire 1487098154 last 1487097702 <3>[603952.700058] LustreError: 15679:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) ### client (nid 1494@gni100) failed to reply to blocking AST (req status 0 rc -5), evict it ns: mdt-atlas1-MDT0000_UUID lock: ffff881c5c9fdd00/0xd94b673bc658b6a7 lrc: 4/0,0 mode: PR/PR res: [0x200259393:0x2574:0x0].0x0 bits 0x13 rrc: 5 type: IBT flags: 0x60200400000020 nid: 1494@gni100 remote: 0x7cf8f23970353e77 expref: 15 pid: 15505 timeout: 4899035982 lvb_type: 0 <3>[603952.745002] LustreError: 15679:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) Skipped 26 previous similar messages <3>[603952.756937] LustreError: 138-a: atlas1-MDT0000: A client on nid 1494@gni100 was evicted due to a lock blocking callback time out: rc -5 <3>[603952.771095] LustreError: Skipped 26 previous similar messages <4>[603952.778032] Lustre: 15679:0:(service.c:2097:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:451s); client may timeout. req@ffff8828331f89c0 x1558697303869196/t646678016128(0) o36->dd20153a-8521-3665-c176-6fbe4f953815@9228@gni100:18/0 lens 616/424 e 12 to 0 dl 1487098603 ref 1 fl Complete:/0/0 rc 0/0 <4>[603952.811716] Lustre: 15679:0:(service.c:2097:ptlrpc_server_handle_request()) Skipped 7 previous similar messages <4>[608212.245234] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[608212.257375] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[608999.939714] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[608999.951864] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[609157.113721] Lustre: atlas1-MDT0000: Connection restored to ac8c1988-3666-be53-f4b6-548b35c5fe95 (at 191@gni4) <4>[609568.501174] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[609568.513321] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[610307.666120] Lustre: atlas1-MDT0000: haven't heard from client 98595f97-05dc-c8b5-48d2-80d5631528a5 (at 8128@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0d88cc00, cur 1487105409 expire 1487104509 last 1487104057 <4>[610337.237572] Lustre: atlas1-MDT0000: Client 9fba0b9f-063b-a24f-5bc8-ceeaa62e8015 (at 10.36.205.208@o2ib) reconnecting <6>[610337.249997] Lustre: atlas1-MDT0000: Connection restored to 9fba0b9f-063b-a24f-5bc8-ceeaa62e8015 (at 10.36.205.208@o2ib) <4>[610449.616319] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[610449.628477] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[610700.205750] Lustre: atlas1-MDT0000: Connection restored to 62de395d-d8ea-2498-0a57-a6150434dcee (at 191@gni4) <4>[611019.537669] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[611019.549887] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[611594.819640] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[611594.831773] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[611623.150107] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[611626.278558] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[611628.597893] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[612196.810957] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[612196.823145] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[617888.587055] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[617888.599459] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[618807.581897] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[618807.594033] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[619376.318854] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[619376.331013] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[620189.106707] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[620189.118836] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[620284.518236] Lustre: atlas1-MDT0000: Connection restored to ace468b8-812f-e1e9-d7c0-c071059e0a50 (at 1336@gni100) <6>[620285.584868] Lustre: atlas1-MDT0000: Connection restored to f38ab4de-7351-d5ce-9cef-6ee7218cec2a (at 5536@gni100) <6>[620285.596800] Lustre: Skipped 7 previous similar messages <6>[620287.744318] Lustre: atlas1-MDT0000: Connection restored to ccc84f14-d033-6f41-2f1c-dfd07833a8f9 (at 14236@gni100) <6>[620287.756343] Lustre: Skipped 3 previous similar messages <6>[620291.874807] Lustre: atlas1-MDT0000: Connection restored to e6a3d813-a2c9-79df-df06-16c834c7136b (at 9621@gni100) <6>[620291.886817] Lustre: Skipped 14 previous similar messages <6>[620299.892495] Lustre: atlas1-MDT0000: Connection restored to 22fd0eb9-7522-f3eb-972d-cba1200bd5de (at 16951@gni100) <6>[620299.904523] Lustre: Skipped 24 previous similar messages <6>[620716.038751] Lustre: atlas1-MDT0000: Connection restored to 68cf9403-2b82-ee03-f45e-65a7ef3828fc (at 16951@gni100) <6>[620716.050785] Lustre: Skipped 19 previous similar messages <4>[620756.722849] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[620756.734994] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[620756.747404] Lustre: Skipped 15 previous similar messages <4>[621046.834634] Lustre: atlas1-MDT0000: Client 9fba0b9f-063b-a24f-5bc8-ceeaa62e8015 (at 10.36.205.208@o2ib) reconnecting <6>[621046.846961] Lustre: atlas1-MDT0000: Connection restored to 9fba0b9f-063b-a24f-5bc8-ceeaa62e8015 (at 10.36.205.208@o2ib) <4>[621325.153745] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[621325.165989] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[621912.738074] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[621912.750248] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[622780.549680] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[622780.562111] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[623139.466682] Lustre: 15928:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1487117673/real 1487117673] req@ffff8807e37e9680 x1558707784495692/t0(0) o104->atlas1-MDT0000@1495@gni100:15/16 lens 296/224 e 0 to 1 dl 1487118240 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[623167.422657] Lustre: 15709:0:(service.c:1336:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[623167.422658] req@ffff8813d40790c0 x1558697308095888/t0(0) o36->dd20153a-8521-3665-c176-6fbe4f953815@9228@gni100:58/0 lens 616/3128 e 12 to 0 dl 1487118273 ref 2 fl Interpret:/0/0 rc 0/0 <4>[623422.464200] Lustre: atlas1-MDT0000: Client dd20153a-8521-3665-c176-6fbe4f953815 (at 9228@gni100) reconnecting <6>[623422.475851] Lustre: atlas1-MDT0000: Connection restored to dd20153a-8521-3665-c176-6fbe4f953815 (at 9228@gni100) <4>[623429.768558] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[623495.423182] Lustre: 15462:0:(service.c:1336:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[623495.423183] req@ffff88283b95a980 x1558697308112132/t0(0) o101->dd20153a-8521-3665-c176-6fbe4f953815@9228@gni100:386/0 lens 696/3384 e 1 to 0 dl 1487118601 ref 2 fl Interpret:/0/0 rc 0/0 <3>[623650.407975] LustreError: 15943:0:(ldlm_request.c:106:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1487118001, 750s ago); not entering recovery in server code, just going back to sleep ns: mdt-atlas1-MDT0000_UUID lock: ffff8839ed57bcc0/0xd94b673dd5413f0f lrc: 3/1,0 mode: --/PR res: [0x200259393:0x2574:0x0].0x0 bits 0x13 rrc: 5 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 15943 timeout: 0 lvb_type: 0 <4>[623706.540102] Lustre: 15928:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1487118240/real 1487118240] req@ffff8807e37e9680 x1558707784495692/t0(0) o104->atlas1-MDT0000@1495@gni100:15/16 lens 296/224 e 0 to 1 dl 1487118807 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 <3>[623706.571437] LustreError: 15928:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) ### client (nid 1495@gni100) failed to reply to blocking AST (req status 0 rc -110), evict it ns: mdt-atlas1-MDT0000_UUID lock: ffff882c5bf3f940/0xd94b673dc62ddbb5 lrc: 4/0,0 mode: PR/PR res: [0x200259393:0x2574:0x0].0x0 bits 0x13 rrc: 5 type: IBT flags: 0x60200400000020 nid: 1495@gni100 remote: 0xffb9e38b51e2143e expref: 13 pid: 16120 timeout: 4918706453 lvb_type: 0 <3>[623706.616583] LustreError: 138-a: atlas1-MDT0000: A client on nid 1495@gni100 was evicted due to a lock blocking callback time out: rc -110 <4>[623706.631076] Lustre: 15943:0:(service.c:2097:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:206s); client may timeout. req@ffff88283b95a980 x1558697308112132/t0(0) o101->dd20153a-8521-3665-c176-6fbe4f953815@9228@gni100:386/0 lens 696/536 e 1 to 0 dl 1487118601 ref 1 fl Complete:/0/0 rc 0/0 <4>[623706.663756] Lustre: 15943:0:(service.c:2097:ptlrpc_server_handle_request()) Skipped 1 previous similar message <4>[624488.944996] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[624488.957271] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[624488.969682] Lustre: Skipped 1 previous similar message <4>[624793.645455] Lustre: atlas1-MDT0000: haven't heard from client eb026d36-1957-79e6-9692-ae47b966c7db (at 5300@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f128ab400, cur 1487119895 expire 1487118995 last 1487118543 <4>[624793.670635] Lustre: Skipped 53 previous similar messages <4>[625065.614096] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[625636.713904] Lustre: atlas1-MDT0000: Client 16e24f2a-a497-51f3-9e69-4b7af20d2cf8 (at 10.36.205.207@o2ib) reconnecting <6>[625636.726228] Lustre: atlas1-MDT0000: Connection restored to 16e24f2a-a497-51f3-9e69-4b7af20d2cf8 (at 10.36.205.207@o2ib) <6>[625636.738820] Lustre: Skipped 1 previous similar message <4>[625645.113271] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[626222.344414] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[626795.455450] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[626795.467621] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[626795.480024] Lustre: Skipped 2 previous similar messages <4>[627395.321604] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[628003.685922] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[628003.698085] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[628003.710493] Lustre: Skipped 1 previous similar message <4>[628575.453917] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[629146.664315] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[629146.676469] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[629146.688866] Lustre: Skipped 1 previous similar message <4>[629745.144316] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[630314.652645] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[630314.664776] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[630314.677203] Lustre: Skipped 1 previous similar message <4>[631461.041218] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[631461.053339] Lustre: Skipped 1 previous similar message <6>[631461.059451] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[631461.071873] Lustre: Skipped 1 previous similar message <4>[632609.031771] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[632609.043886] Lustre: Skipped 1 previous similar message <6>[632609.050001] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[632609.062429] Lustre: Skipped 1 previous similar message <4>[632920.639498] Lustre: atlas1-MDT0000: haven't heard from client ddb3405b-b312-ca9e-fd47-7c5e38e09c06 (at 7792@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f11a9b400, cur 1487128022 expire 1487127122 last 1487126670 <4>[633421.772692] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[633421.784808] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[634573.632270] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[634573.644384] Lustre: Skipped 2 previous similar messages <6>[634573.650666] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[634573.663077] Lustre: Skipped 2 previous similar messages <4>[635713.708901] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[635713.721016] Lustre: Skipped 1 previous similar message <6>[635713.727088] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[635713.739524] Lustre: Skipped 1 previous similar message <4>[636858.321319] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[636858.333438] Lustre: Skipped 1 previous similar message <6>[636858.339547] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[636858.352069] Lustre: Skipped 1 previous similar message <4>[638012.098904] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[638012.111006] Lustre: Skipped 2 previous similar messages <6>[638012.117289] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[638012.129703] Lustre: Skipped 2 previous similar messages <4>[639162.840115] Lustre: atlas1-MDT0000: Client 16e24f2a-a497-51f3-9e69-4b7af20d2cf8 (at 10.36.205.207@o2ib) reconnecting <4>[639162.852427] Lustre: Skipped 1 previous similar message <6>[639162.858511] Lustre: atlas1-MDT0000: Connection restored to 16e24f2a-a497-51f3-9e69-4b7af20d2cf8 (at 10.36.205.207@o2ib) <6>[639162.871111] Lustre: Skipped 1 previous similar message <4>[640303.406134] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[640303.418269] Lustre: Skipped 2 previous similar messages <6>[640303.424461] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[640303.436879] Lustre: Skipped 2 previous similar messages <4>[641440.010574] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[641440.022707] Lustre: Skipped 1 previous similar message <6>[641440.028796] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[641440.041192] Lustre: Skipped 1 previous similar message <4>[642583.165093] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[642583.177205] Lustre: Skipped 1 previous similar message <6>[642583.183305] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[642583.195756] Lustre: Skipped 1 previous similar message <4>[643420.620394] Lustre: atlas1-MDT0000: haven't heard from client b6adb02d-3c10-1ccd-17bd-21deb9a8b10a (at 91@gni4) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0ecf6000, cur 1487138522 expire 1487137622 last 1487137170 <4>[643743.341640] Lustre: atlas1-MDT0000: Client 16e24f2a-a497-51f3-9e69-4b7af20d2cf8 (at 10.36.205.207@o2ib) reconnecting <4>[643743.353984] Lustre: Skipped 1 previous similar message <6>[643743.360071] Lustre: atlas1-MDT0000: Connection restored to 16e24f2a-a497-51f3-9e69-4b7af20d2cf8 (at 10.36.205.207@o2ib) <6>[643743.372680] Lustre: Skipped 1 previous similar message <4>[644906.522704] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[644906.534825] Lustre: Skipped 2 previous similar messages <6>[644906.541016] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[644906.553430] Lustre: Skipped 2 previous similar messages <4>[644907.618254] Lustre: atlas1-MDT0000: haven't heard from client 342b2135-8a27-2bf3-05bf-9ab4f1fb34a4 (at 13256@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881c62d39000, cur 1487140009 expire 1487139109 last 1487138657 <4>[644907.643545] Lustre: Skipped 168 previous similar messages <4>[646054.163318] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[646054.175431] Lustre: Skipped 2 previous similar messages <6>[646054.181609] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[646054.194007] Lustre: Skipped 2 previous similar messages <6>[647176.327861] Lustre: atlas1-MDT0000: Connection restored to 20306928-c577-eec7-9593-d460d7352989 (at 17@gni4) <6>[647176.339448] Lustre: Skipped 1 previous similar message <4>[647433.101462] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[647433.113593] Lustre: Skipped 1 previous similar message <4>[647637.614812] Lustre: atlas1-MDT0000: haven't heard from client 7cfbbba0-fbe4-43ad-4573-ce95d9bf7ea5 (at 18140@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f11379400, cur 1487142739 expire 1487141839 last 1487141387 <6>[647811.431751] Lustre: atlas1-MDT0000: Connection restored to adf17aa3-aca8-71ac-4375-cca5d6d01a16 (at 129@gni4) <6>[647811.443415] Lustre: Skipped 1 previous similar message <4>[648021.693340] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[648343.348848] Lustre: atlas1-MDT0000: Client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) reconnecting <6>[648594.252351] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[648594.264749] Lustre: Skipped 3 previous similar messages <4>[648986.135612] Lustre: atlas1-MDT0000: Client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) reconnecting <4>[648986.147025] Lustre: Skipped 2 previous similar messages <3>[649185.832932] LustreError: 15773:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20038231d:0xca8:0x0]: rc = -2 <4>[649735.142621] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[649735.154735] Lustre: Skipped 1 previous similar message <6>[649735.160812] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[649735.173219] Lustre: Skipped 2 previous similar messages <4>[650350.646887] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[650350.658997] Lustre: Skipped 3 previous similar messages <6>[650350.665180] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[650350.677596] Lustre: Skipped 3 previous similar messages <4>[651390.237825] Lustre: atlas1-MDT0000: Client 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) reconnecting <4>[651390.249181] Lustre: Skipped 2 previous similar messages <6>[651390.255387] Lustre: atlas1-MDT0000: Connection restored to 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) <6>[651390.267020] Lustre: Skipped 2 previous similar messages <4>[652077.383713] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[652077.395825] Lustre: Skipped 1 previous similar message <6>[652077.401919] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[652077.414321] Lustre: Skipped 1 previous similar message <4>[652233.609695] Lustre: atlas1-MDT0000: haven't heard from client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883b02e3d400, cur 1487147335 expire 1487146435 last 1487145983 <4>[653140.307950] Lustre: atlas1-MDT0000: Client 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) reconnecting <4>[653140.319307] Lustre: Skipped 1 previous similar message <6>[653140.325386] Lustre: atlas1-MDT0000: Connection restored to 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) <6>[653140.337030] Lustre: Skipped 1 previous similar message <4>[653811.445677] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[653811.457882] Lustre: Skipped 2 previous similar messages <6>[653811.464056] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[653811.476507] Lustre: Skipped 3 previous similar messages <4>[654879.418288] Lustre: atlas1-MDT0000: Client 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) reconnecting <4>[654879.429638] Lustre: Skipped 1 previous similar message <6>[654879.435776] Lustre: atlas1-MDT0000: Connection restored to adf17aa3-aca8-71ac-4375-cca5d6d01a16 (at 129@gni4) <6>[654879.447465] Lustre: Skipped 1 previous similar message <4>[655038.604369] Lustre: atlas1-MDT0000: haven't heard from client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) in 1201 seconds. I think it's dead, and I am evicting it. exp ffff8804581cd800, cur 1487150140 expire 1487149240 last 1487148939 <4>[655529.726786] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[655529.738930] Lustre: Skipped 2 previous similar messages <6>[655529.745133] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[655529.757535] Lustre: Skipped 2 previous similar messages <4>[656273.390184] Lustre: atlas1-MDT0000: Client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) reconnecting <4>[656273.401453] Lustre: Skipped 2 previous similar messages <6>[656273.407627] Lustre: atlas1-MDT0000: Connection restored to e0988e3d-a9e2-441f-7c5d-cca8e05ebb7d (at 17@gni4) <6>[656273.419231] Lustre: Skipped 3 previous similar messages <4>[656890.536986] Lustre: atlas1-MDT0000: Client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) reconnecting <4>[656890.548222] Lustre: Skipped 2 previous similar messages <6>[656890.554405] Lustre: atlas1-MDT0000: Connection restored to e0988e3d-a9e2-441f-7c5d-cca8e05ebb7d (at 17@gni4) <6>[656890.565937] Lustre: Skipped 2 previous similar messages <4>[657507.580720] Lustre: atlas1-MDT0000: Client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) reconnecting <4>[657507.591961] Lustre: Skipped 2 previous similar messages <6>[657507.598140] Lustre: atlas1-MDT0000: Connection restored to 20306928-c577-eec7-9593-d460d7352989 (at 17@gni4) <6>[657507.609801] Lustre: Skipped 2 previous similar messages <4>[658440.884888] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[658440.897015] Lustre: Skipped 1 previous similar message <6>[658440.903092] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[658440.915499] Lustre: Skipped 1 previous similar message <4>[659091.578503] Lustre: atlas1-MDT0000: Client 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) reconnecting <4>[659091.589871] Lustre: Skipped 2 previous similar messages <6>[659091.596048] Lustre: atlas1-MDT0000: Connection restored to adf17aa3-aca8-71ac-4375-cca5d6d01a16 (at 129@gni4) <6>[659091.607680] Lustre: Skipped 2 previous similar messages <4>[659258.599762] Lustre: atlas1-MDT0000: haven't heard from client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) in 1208 seconds. I think it's dead, and I am evicting it. exp ffff883f0f000400, cur 1487154360 expire 1487153460 last 1487153152 <4>[659782.685091] Lustre: atlas1-MDT0000: Client 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) reconnecting <4>[659782.696434] Lustre: Skipped 1 previous similar message <6>[659782.702549] Lustre: atlas1-MDT0000: Connection restored to adf17aa3-aca8-71ac-4375-cca5d6d01a16 (at 129@gni4) <6>[659782.714202] Lustre: Skipped 1 previous similar message <6>[660465.331133] Lustre: atlas1-MDT0000: Connection restored to 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) <6>[660465.342671] Lustre: Skipped 2 previous similar messages <4>[660750.121791] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[660750.133909] Lustre: Skipped 2 previous similar messages <6>[661347.903977] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[661347.916391] Lustre: Skipped 1 previous similar message <4>[661676.596216] Lustre: atlas1-MDT0000: haven't heard from client 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0ac94400, cur 1487156778 expire 1487155878 last 1487155426 <4>[661918.407750] Lustre: atlas1-MDT0000: Client 16e24f2a-a497-51f3-9e69-4b7af20d2cf8 (at 10.36.205.207@o2ib) reconnecting <4>[661918.420063] Lustre: Skipped 1 previous similar message <4>[662127.594668] Lustre: atlas1-MDT0000: haven't heard from client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) in 1088 seconds. I think it's dead, and I am evicting it. exp ffff8813dbd48c00, cur 1487157229 expire 1487156329 last 1487156141 <6>[662166.925168] Lustre: atlas1-MDT0000: Connection restored to adf17aa3-aca8-71ac-4375-cca5d6d01a16 (at 129@gni4) <6>[662166.936800] Lustre: Skipped 2 previous similar messages <4>[662854.994796] Lustre: atlas1-MDT0000: Client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) reconnecting <4>[662855.006035] Lustre: Skipped 2 previous similar messages <6>[662855.012210] Lustre: atlas1-MDT0000: Connection restored to 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) <6>[662855.023733] Lustre: Skipped 2 previous similar messages <4>[663472.146556] Lustre: atlas1-MDT0000: Client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) reconnecting <4>[663472.157790] Lustre: Skipped 2 previous similar messages <6>[663472.163985] Lustre: atlas1-MDT0000: Connection restored to 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) <6>[663472.175513] Lustre: Skipped 2 previous similar messages <4>[664118.194250] Lustre: atlas1-MDT0000: Client 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) reconnecting <4>[664118.205578] Lustre: Skipped 1 previous similar message <6>[664118.211671] Lustre: atlas1-MDT0000: Connection restored to 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) <6>[664118.223299] Lustre: Skipped 1 previous similar message <4>[664785.231115] Lustre: atlas1-MDT0000: Client 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) reconnecting <4>[664785.242470] Lustre: Skipped 2 previous similar messages <6>[664785.248674] Lustre: atlas1-MDT0000: Connection restored to f6632a46-c101-bb11-4454-563d80063513 (at 129@gni4) <6>[664785.260296] Lustre: Skipped 2 previous similar messages <4>[665395.475460] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[665395.487599] Lustre: Skipped 1 previous similar message <6>[665395.493679] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[665395.506094] Lustre: Skipped 1 previous similar message <4>[666003.138634] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[666003.150755] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[666120.589070] Lustre: atlas1-MDT0000: haven't heard from client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8821a5ceb000, cur 1487161222 expire 1487160322 last 1487159870 <4>[666571.589295] Lustre: atlas1-MDT0000: haven't heard from client 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) in 1216 seconds. I think it's dead, and I am evicting it. exp ffff883dd7d87400, cur 1487161673 expire 1487160773 last 1487160457 <4>[667152.108128] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[667152.120235] Lustre: Skipped 1 previous similar message <6>[667152.126329] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[667152.138735] Lustre: Skipped 1 previous similar message <4>[668291.074645] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[668291.086788] Lustre: Skipped 1 previous similar message <6>[668291.092896] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[668291.105330] Lustre: Skipped 1 previous similar message <4>[668909.630083] Lustre: atlas1-MDT0000: Client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) reconnecting <4>[668909.641379] Lustre: Skipped 1 previous similar message <6>[668909.647486] Lustre: atlas1-MDT0000: Connection restored to e0988e3d-a9e2-441f-7c5d-cca8e05ebb7d (at 17@gni4) <6>[668909.659022] Lustre: Skipped 3 previous similar messages <4>[669222.711738] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[669225.647536] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[669227.695454] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[669826.591164] Lustre: atlas1-MDT0000: haven't heard from client 9ea594a1-adc3-562a-bb8b-eaf1644960ce (at 11855@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0b270000, cur 1487164928 expire 1487164028 last 1487163576 <4>[670011.902714] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[670011.914831] Lustre: Skipped 2 previous similar messages <6>[670011.921046] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[670011.933457] Lustre: Skipped 2 previous similar messages <4>[670637.924208] Lustre: atlas1-MDT0000: Client 928ae266-a56b-07ed-20aa-45ae48311726 (at 1830@gni100) reconnecting <4>[670637.935865] Lustre: Skipped 4 previous similar messages <6>[670637.942035] Lustre: atlas1-MDT0000: Connection restored to 928ae266-a56b-07ed-20aa-45ae48311726 (at 1830@gni100) <6>[670637.953976] Lustre: Skipped 4 previous similar messages <4>[670926.583478] Lustre: atlas1-MDT0000: haven't heard from client 3eaf9d1d-e8db-fc4d-8cfc-9a62326b4540 (at 17949@gni100) in 1254 seconds. I think it's dead, and I am evicting it. exp ffff883f0af0c800, cur 1487166028 expire 1487165128 last 1487164774 <4>[670926.608753] Lustre: Skipped 245 previous similar messages <4>[671384.168552] Lustre: atlas1-MDT0000: Client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) reconnecting <4>[671384.184958] Lustre: Skipped 1057 previous similar messages <6>[671384.191432] Lustre: atlas1-MDT0000: Connection restored to e0988e3d-a9e2-441f-7c5d-cca8e05ebb7d (at 17@gni4) <6>[671384.203001] Lustre: Skipped 1057 previous similar messages <4>[672313.915518] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[672313.927668] Lustre: Skipped 1 previous similar message <6>[672313.933743] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[672313.946174] Lustre: Skipped 2 previous similar messages <4>[672424.582218] Lustre: atlas1-MDT0000: haven't heard from client 402596ba-563e-6c2e-4208-e96bf91df171 (at 12411@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0f981000, cur 1487167526 expire 1487166626 last 1487166174 <4>[672424.607575] Lustre: Skipped 1 previous similar message <4>[672916.437139] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[672916.449268] Lustre: Skipped 2 previous similar messages <6>[672916.455495] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[672916.467909] Lustre: Skipped 2 previous similar messages <3>[673504.692681] LustreError: 16484:0:(ldlm_lib.c:3122:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff883ed5066050 x1558686925534964/t0(0) o37->49207f95-72eb-913e-ab93-3a579be04014@10.36.225.5@o2ib:123/0 lens 568/440 e 0 to 0 dl 1487168923 ref 1 fl Interpret:/0/0 rc 0/0 <3>[673843.328407] LustreError: 15722:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20038b5ab:0x3f83:0x0]: rc = -2 <4>[673894.849422] Lustre: atlas1-MDT0000: Client 41757a56-4026-e22f-97ab-61970d8ebcd9 (at 14964@gni100) reconnecting <4>[673894.861190] Lustre: Skipped 1 previous similar message <6>[673894.867257] Lustre: atlas1-MDT0000: Connection restored to 41757a56-4026-e22f-97ab-61970d8ebcd9 (at 14964@gni100) <6>[673894.879260] Lustre: Skipped 1 previous similar message <4>[674662.463418] Lustre: atlas1-MDT0000: Client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) reconnecting <4>[674662.474684] Lustre: Skipped 4 previous similar messages <6>[674662.480863] Lustre: atlas1-MDT0000: Connection restored to e0988e3d-a9e2-441f-7c5d-cca8e05ebb7d (at 17@gni4) <6>[674662.492481] Lustre: Skipped 4 previous similar messages <4>[674766.595172] Lustre: atlas1-MDT0000: haven't heard from client 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0abfdc00, cur 1487169868 expire 1487168968 last 1487168516 <4>[675278.652549] Lustre: atlas1-MDT0000: Client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) reconnecting <4>[675278.663797] Lustre: Skipped 2 previous similar messages <6>[675278.669975] Lustre: atlas1-MDT0000: Connection restored to e0988e3d-a9e2-441f-7c5d-cca8e05ebb7d (at 17@gni4) <6>[675278.681504] Lustre: Skipped 3 previous similar messages <4>[675939.231945] Lustre: atlas1-MDT0000: Client 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) reconnecting <4>[675939.243284] Lustre: Skipped 1 previous similar message <6>[675939.249361] Lustre: atlas1-MDT0000: Connection restored to a3af9b0b-c641-f1d1-5a88-1f8010986ea1 (at 129@gni4) <6>[675939.260985] Lustre: Skipped 1 previous similar message <4>[676606.208323] Lustre: atlas1-MDT0000: Client 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) reconnecting <4>[676606.219698] Lustre: Skipped 2 previous similar messages <6>[676606.225884] Lustre: atlas1-MDT0000: Connection restored to 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) <6>[676606.237510] Lustre: Skipped 2 previous similar messages <4>[677453.388442] Lustre: atlas1-MDT0000: Client 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) reconnecting <4>[677453.399818] Lustre: Skipped 2 previous similar messages <6>[677453.406009] Lustre: atlas1-MDT0000: Connection restored to adf17aa3-aca8-71ac-4375-cca5d6d01a16 (at 129@gni4) <6>[677453.417677] Lustre: Skipped 2 previous similar messages <4>[678095.436038] Lustre: atlas1-MDT0000: Client 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) reconnecting <4>[678095.447402] Lustre: Skipped 1 previous similar message <6>[678095.453503] Lustre: atlas1-MDT0000: Connection restored to f6632a46-c101-bb11-4454-563d80063513 (at 129@gni4) <6>[678095.465266] Lustre: Skipped 1 previous similar message <4>[678817.348744] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[678817.360871] Lustre: Skipped 3 previous similar messages <6>[678817.367064] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[678817.379463] Lustre: Skipped 3 previous similar messages <4>[679967.502980] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[679967.515101] Lustre: Skipped 2 previous similar messages <6>[679967.521274] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[679967.533702] Lustre: Skipped 2 previous similar messages <4>[680697.029944] Lustre: atlas1-MDT0000: Client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) reconnecting <4>[680697.041179] Lustre: Skipped 3 previous similar messages <6>[680697.047359] Lustre: atlas1-MDT0000: Connection restored to 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) <6>[680697.058890] Lustre: Skipped 3 previous similar messages <4>[681742.885206] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[681742.897339] Lustre: Skipped 2 previous similar messages <6>[681742.903643] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[681742.916053] Lustre: Skipped 2 previous similar messages <4>[682615.567091] Lustre: atlas1-MDT0000: haven't heard from client 20207a2a-0930-ee28-62df-e9e4d0e68f48 (at 17@gni4) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff882178038000, cur 1487177717 expire 1487176817 last 1487176365 <4>[682945.344275] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[682945.356391] Lustre: Skipped 1 previous similar message <6>[682945.362470] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[682945.374874] Lustre: Skipped 1 previous similar message <4>[683066.566491] Lustre: atlas1-MDT0000: haven't heard from client 65928f93-6572-0cf4-fc7b-c0b95396ece7 (at 129@gni4) in 1302 seconds. I think it's dead, and I am evicting it. exp ffff88389de25c00, cur 1487178168 expire 1487177268 last 1487176866 <4>[683619.599685] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[683622.616997] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[683624.709410] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[684146.542405] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[684146.554525] Lustre: Skipped 1 previous similar message <6>[684146.560759] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[684146.573169] Lustre: Skipped 1 previous similar message <4>[684916.231920] Lustre: atlas1-MDT0000: Client 10d8bf25-0a1c-19a9-4491-2e841f0d5b8d (at 17@gni4) reconnecting <4>[684916.243172] Lustre: Skipped 1 previous similar message <6>[684916.249286] Lustre: atlas1-MDT0000: Connection restored to 585236fb-f56a-2fed-abe5-fdba01d0a69f (at 17@gni4) <6>[684916.260827] Lustre: Skipped 3 previous similar messages <4>[685626.521880] Lustre: atlas1-MDT0000: Client 10d8bf25-0a1c-19a9-4491-2e841f0d5b8d (at 17@gni4) reconnecting <4>[685626.533123] Lustre: Skipped 2 previous similar messages <6>[685626.539298] Lustre: atlas1-MDT0000: Connection restored to 10d8bf25-0a1c-19a9-4491-2e841f0d5b8d (at 17@gni4) <6>[685626.550840] Lustre: Skipped 2 previous similar messages <6>[686488.294219] Lustre: atlas1-MDT0000: Connection restored to 24cd3924-4666-e672-e3c9-a1ebd16a1b80 (at 191@gni4) <6>[686488.305862] Lustre: Skipped 2 previous similar messages <4>[686834.848517] Lustre: atlas1-MDT0000: Client 10d8bf25-0a1c-19a9-4491-2e841f0d5b8d (at 17@gni4) reconnecting <4>[686834.859766] Lustre: Skipped 2 previous similar messages <6>[687109.627305] Lustre: atlas1-MDT0000: Connection restored to adf17aa3-aca8-71ac-4375-cca5d6d01a16 (at 129@gni4) <6>[687109.638944] Lustre: Skipped 2 previous similar messages <4>[687435.647034] Lustre: atlas1-MDT0000: Client 10d8bf25-0a1c-19a9-4491-2e841f0d5b8d (at 17@gni4) reconnecting <4>[687435.658283] Lustre: Skipped 2 previous similar messages <6>[687726.752115] Lustre: atlas1-MDT0000: Connection restored to 398a21fe-38d8-d005-7451-b0779a3dcc9e (at 129@gni4) <6>[687726.763744] Lustre: Skipped 2 previous similar messages <4>[688052.680131] Lustre: atlas1-MDT0000: Client 10d8bf25-0a1c-19a9-4491-2e841f0d5b8d (at 17@gni4) reconnecting <4>[688052.691378] Lustre: Skipped 2 previous similar messages <6>[688368.879241] Lustre: atlas1-MDT0000: Connection restored to 398a21fe-38d8-d005-7451-b0779a3dcc9e (at 129@gni4) <6>[688368.890873] Lustre: Skipped 4 previous similar messages <4>[688682.002113] Lustre: atlas1-MDT0000: Client a5b037db-d604-5787-1847-d86c1486a45d (at 190@gni4) reconnecting <4>[688682.013461] Lustre: Skipped 5 previous similar messages <6>[689190.340161] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[689190.352572] Lustre: Skipped 3 previous similar messages <4>[689758.414259] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[689758.426378] Lustre: Skipped 2 previous similar messages <6>[689997.954807] Lustre: atlas1-MDT0000: Connection restored to a5b037db-d604-5787-1847-d86c1486a45d (at 190@gni4) <6>[689997.966440] Lustre: Skipped 2 previous similar messages <4>[690894.454934] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[690894.467056] Lustre: Skipped 2 previous similar messages <6>[690894.473332] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[690894.485742] Lustre: Skipped 1 previous similar message <4>[691745.674724] Lustre: atlas1-MDT0000: Client b3c5618a-c333-0113-1a4b-1c5951f265d5 (at 10.36.205.218@o2ib) reconnecting <4>[691745.687046] Lustre: Skipped 2 previous similar messages <6>[691745.693230] Lustre: atlas1-MDT0000: Connection restored to b3c5618a-c333-0113-1a4b-1c5951f265d5 (at 10.36.205.218@o2ib) <6>[691745.705849] Lustre: Skipped 168 previous similar messages <4>[692443.665592] Lustre: atlas1-MDT0000: Client b3c5618a-c333-0113-1a4b-1c5951f265d5 (at 10.36.205.218@o2ib) reconnecting <4>[692443.677909] Lustre: Skipped 2 previous similar messages <6>[692443.689378] Lustre: atlas1-MDT0000: Connection restored to b3c5618a-c333-0113-1a4b-1c5951f265d5 (at 10.36.205.218@o2ib) <6>[692443.702005] Lustre: Skipped 2 previous similar messages <4>[693094.828574] Lustre: atlas1-MDT0000: Client e8de95e8-c0bd-3bc7-7904-7130e31ec181 (at 10.36.205.217@o2ib) reconnecting <4>[693094.840895] Lustre: Skipped 3 previous similar messages <6>[693094.847205] Lustre: atlas1-MDT0000: Connection restored to e8de95e8-c0bd-3bc7-7904-7130e31ec181 (at 10.36.205.217@o2ib) <6>[693094.859821] Lustre: Skipped 3 previous similar messages <4>[693737.435029] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[693737.447154] Lustre: Skipped 2 previous similar messages <6>[693737.453329] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[693737.465736] Lustre: Skipped 2 previous similar messages <4>[694618.943311] Lustre: atlas1-MDT0000: Client 9df83e43-b6c5-89ff-aadc-8e3a70ca343b (at 10.36.205.199@o2ib) reconnecting <4>[694618.955634] Lustre: Skipped 2 previous similar messages <6>[694618.961817] Lustre: atlas1-MDT0000: Connection restored to 9df83e43-b6c5-89ff-aadc-8e3a70ca343b (at 10.36.205.199@o2ib) <6>[694618.974418] Lustre: Skipped 2 previous similar messages <4>[695398.361695] Lustre: atlas1-MDT0000: Client e8de95e8-c0bd-3bc7-7904-7130e31ec181 (at 10.36.205.217@o2ib) reconnecting <4>[695398.374012] Lustre: Skipped 3 previous similar messages <6>[695398.380343] Lustre: atlas1-MDT0000: Connection restored to e8de95e8-c0bd-3bc7-7904-7130e31ec181 (at 10.36.205.217@o2ib) <6>[695398.392945] Lustre: Skipped 251 previous similar messages <4>[696010.657169] Lustre: atlas1-MDT0000: Client e8de95e8-c0bd-3bc7-7904-7130e31ec181 (at 10.36.205.217@o2ib) reconnecting <4>[696010.669486] Lustre: Skipped 1 previous similar message <6>[696010.675644] Lustre: atlas1-MDT0000: Connection restored to e8de95e8-c0bd-3bc7-7904-7130e31ec181 (at 10.36.205.217@o2ib) <6>[696010.688248] Lustre: Skipped 1 previous similar message <4>[697170.082797] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[697170.094918] Lustre: Skipped 2 previous similar messages <6>[697170.101121] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[697170.113660] Lustre: Skipped 186 previous similar messages <4>[697467.545420] Lustre: atlas1-MDT0000: haven't heard from client c4f0b6fe-6bea-f62b-dcd3-274199c41e28 (at 10967@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f12e32800, cur 1487192569 expire 1487191669 last 1487191217 <4>[698020.844052] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[698023.858133] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[698025.929036] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <6>[698077.521447] Lustre: atlas1-MDT0000: Connection restored to c41dd2a8-b812-3989-0925-bd37cd659710 (at 7240@gni100) <6>[698077.533462] Lustre: Skipped 1 previous similar message <4>[698315.124525] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[698315.136647] Lustre: Skipped 1 previous similar message <6>[698891.449157] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[698891.461660] Lustre: Skipped 181 previous similar messages <4>[699461.162567] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[699461.174687] Lustre: Skipped 1 previous similar message <6>[700037.823286] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[700037.835696] Lustre: Skipped 1 previous similar message <3>[700370.862914] LustreError: 15643:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x200378e60:0x88b:0x0]: rc = -2 <4>[700609.834715] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[700609.846841] Lustre: Skipped 1 previous similar message <3>[701027.512525] LustreError: 16123:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20039f714:0x4ad4:0x0]: rc = -2 <3>[701029.107840] LustreError: 15494:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20039f724:0x48c5:0x0]: rc = -2 <6>[701180.972898] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[701180.985309] Lustre: Skipped 1 previous similar message <4>[702342.459763] Lustre: atlas1-MDT0000: Client cd3a4330-f292-099a-3d2e-48127a39339c (at 7515@gni100) reconnecting <6>[702342.470738] Lustre: atlas1-MDT0000: Connection restored to 41093801-e378-4d19-21d6-37f97c64faa8 (at 9688@gni100) <4>[702342.483414] Lustre: Skipped 2 previous similar messages <4>[702967.579137] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[702967.591290] Lustre: Skipped 12 previous similar messages <6>[702967.597692] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[702967.610111] Lustre: Skipped 13 previous similar messages <4>[703537.976247] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[704126.533330] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[704126.545479] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[704126.557944] Lustre: Skipped 1 previous similar message <4>[705273.270852] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[705273.282991] Lustre: Skipped 1 previous similar message <6>[705273.289064] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[705273.301556] Lustre: Skipped 1 previous similar message <4>[706447.942311] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[706447.954433] Lustre: Skipped 1 previous similar message <6>[706447.960534] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[706447.972935] Lustre: Skipped 1 previous similar message <4>[707606.285845] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[707606.297959] Lustre: Skipped 1 previous similar message <6>[707606.304040] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[707606.316529] Lustre: Skipped 1 previous similar message <4>[708747.152626] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[708747.164746] Lustre: Skipped 1 previous similar message <6>[708747.170931] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[708747.183343] Lustre: Skipped 1 previous similar message <4>[709899.691146] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[709899.703285] Lustre: Skipped 1 previous similar message <6>[709899.709390] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[709899.721841] Lustre: Skipped 1 previous similar message <4>[711039.302756] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[711039.314886] Lustre: Skipped 1 previous similar message <6>[711039.320987] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[711039.333409] Lustre: Skipped 1 previous similar message <6>[711648.418147] Lustre: atlas1-MDT0000: Connection restored to 225d92bc-cd09-4c21-3c0b-03211e7e3063 (at 16466@gni100) <6>[711648.430175] Lustre: Skipped 5 previous similar messages <4>[712199.372215] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[712199.384334] Lustre: Skipped 1 previous similar message <6>[712767.799537] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[712767.811941] Lustre: Skipped 52 previous similar messages <4>[713336.310753] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[713336.322876] Lustre: Skipped 1 previous similar message <6>[713536.871107] Lustre: atlas1-MDT0000: Connection restored to f4c6d58a-5893-c4ae-6839-30ffd023f4b7 (at 2836@gni100) <6>[713536.883055] Lustre: Skipped 10 previous similar messages <6>[714205.491772] Lustre: atlas1-MDT0000: Connection restored to d95bb330-2475-d0b9-8226-b5403d53bb52 (at 13901@gni100) <6>[714205.503792] Lustre: Skipped 1 previous similar message <4>[714635.951209] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[714635.963338] Lustre: Skipped 1 previous similar message <4>[715272.948387] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[715272.960518] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[715272.972948] Lustre: Skipped 3 previous similar messages <4>[715895.520015] Lustre: atlas1-MDT0000: haven't heard from client 76738d5d-1d68-aa21-a2b7-7dbdf2610aab (at 11693@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0cb80c00, cur 1487210997 expire 1487210097 last 1487209645 <4>[715895.545307] Lustre: Skipped 225 previous similar messages <4>[716105.401260] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[716105.413393] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[716741.184299] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[716741.196514] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[717309.224672] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[718200.838416] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[718200.850551] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[718200.862950] Lustre: Skipped 1 previous similar message <4>[718897.458739] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[718897.470892] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[720053.864696] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[720053.876826] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[720827.956390] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[720827.968535] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[722321.978510] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[722321.990632] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[722908.933511] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[722908.945626] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[723559.250756] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[723559.262910] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[724352.630501] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[724352.642641] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[724759.523313] Lustre: atlas1-MDT0000: haven't heard from client 5885be20-5a89-bd7c-8d9f-37553f16e0d0 (at 7539@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0ecbf000, cur 1487219861 expire 1487218961 last 1487218509 <4>[724967.758480] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[724967.770613] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[725719.718354] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[725719.730483] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[727041.258578] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[727041.270704] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[727740.638682] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[727740.650843] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[729037.927098] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[729037.939240] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[729776.352965] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[729776.365129] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[730695.396611] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[730695.408731] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[731552.964582] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[731552.976707] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[732149.662867] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[732149.675084] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[732910.895637] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[732910.907761] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[733583.401797] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[733583.413929] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[734247.565786] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[734247.577924] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[734896.146922] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[734896.159070] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[735527.668998] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[735527.681140] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <3>[735999.889488] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 14087@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff883df6a733c0/0xd94b6743671138ef lrc: 4/0,0 mode: PR/PR res: [0x2002e1c31:0x1d947:0x0].0x0 bits 0x13 rrc: 5 type: IBT flags: 0x60200400000020 nid: 14087@gni100 remote: 0x95720ccad264685a expref: 22 pid: 15540 timeout: 5030666572 lvb_type: 0 <3>[735999.932867] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) Skipped 17 previous similar messages <3>[735999.944503] LustreError: 15720:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) ### client (nid 14087@gni100) failed to reply to blocking AST (req status 0 rc -5), evict it ns: mdt-atlas1-MDT0000_UUID lock: ffff883df6a733c0/0xd94b6743671138ef lrc: 4/0,0 mode: PR/PR res: [0x2002e1c31:0x1d947:0x0].0x0 bits 0x13 rrc: 5 type: IBT flags: 0x60200400000020 nid: 14087@gni100 remote: 0x95720ccad264685a expref: 22 pid: 15540 timeout: 5030666572 lvb_type: 0 <3>[735999.989736] LustreError: 138-a: atlas1-MDT0000: A client on nid 14087@gni100 was evicted due to a lock blocking callback time out: rc -5 <4>[736279.981926] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[736279.994056] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[737233.356937] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[737233.369074] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[737820.354011] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[737820.366159] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[738391.489527] Lustre: atlas1-MDT0000: haven't heard from client fbdb863d-a118-6e15-54a1-14bb302b43d7 (at 17456@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0db45c00, cur 1487233493 expire 1487232593 last 1487232141 <4>[738508.772156] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[738508.784302] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[739098.179454] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[739098.191582] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[739677.247717] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[739677.259865] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[740321.425925] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[740321.438090] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[741036.525888] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[741036.538007] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[741741.976931] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[741741.989079] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[742696.107768] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[742696.119892] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[745403.872911] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[745403.885090] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[746911.353482] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[746911.365626] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[747507.201456] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[747507.213587] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[748196.353211] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[748196.365352] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[748774.254390] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[748774.266517] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[749471.401237] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[749471.413356] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[749493.474202] Lustre: atlas1-MDT0000: haven't heard from client e35f44c1-d0cd-de87-5084-cad5d666213f (at 224@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f10d67c00, cur 1487244595 expire 1487243695 last 1487243243 <4>[750072.272358] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[750072.284485] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[750769.793166] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[750769.805291] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[751602.135709] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[751602.147850] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[752441.768247] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[752441.780372] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[753102.097305] Lustre: atlas1-MDT0000: Connection restored to 10.36.226.117@o2ib (at 0@lo) <6>[753102.106846] Lustre: client wants to enable acl, but mdt not! <4>[753102.290998] Lustre: Mounted atlas1-client <4>[753138.489404] telegraf: page allocation failure. order:1, mode:0x20 <4>[753138.496540] Pid: 14453, comm: telegraf Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[753138.505826] Call Trace: <4>[753138.508877] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[753138.516499] [] ? number+0x1f0/0x320 <4>[753138.522571] [] ? kmem_getpages+0x62/0x170 <4>[753138.529220] [] ? fallback_alloc+0x1ba/0x270 <4>[753138.536065] [] ? cache_grow+0x2cf/0x320 <4>[753138.542523] [] ? ____cache_alloc_node+0x99/0x160 <4>[753138.549875] [] ? lprocfs_stats_alloc_one+0x84/0x360 [obdclass] <4>[753138.558785] [] ? __kmalloc+0x199/0x230 <4>[753138.565162] [] ? lprocfs_stats_alloc_one+0x84/0x360 [obdclass] <4>[753138.574084] [] ? lprocfs_counter_add+0x1a8/0x1c0 [obdclass] <4>[753138.582733] [] ? ptl_send_rpc+0x5be/0xea0 [ptlrpc] <4>[753138.590289] [] ? ptlrpc_send_new_req+0x50e/0x9b0 [ptlrpc] <4>[753138.598533] [] ? ptlrpc_set_wait+0x6a6/0x960 [ptlrpc] <4>[753138.606350] [] ? osc_statfs_async+0xcc/0x210 [osc] <4>[753138.613881] [] ? lov_statfs_async+0x15b/0x670 [lov] <4>[753138.621517] [] ? ll_statfs_internal+0x4c5/0xcb0 [lustre] <4>[753138.629623] [] ? mntput_no_expire+0x30/0x110 <4>[753138.636568] [] ? ll_statfs+0x95/0x190 [lustre] <4>[753138.643704] [] ? statfs_by_dentry+0x74/0xa0 <4>[753138.650547] [] ? vfs_statfs+0x1b/0xb0 <4>[753138.656810] [] ? user_statfs+0x47/0xb0 <4>[753138.663171] [] ? sys_statfs+0x2a/0x50 <4>[753138.669431] [] ? system_call_fastpath+0x16/0x1b <6>[753138.676662] Mem-Info: <4>[753138.679521] Node 0 DMA per-cpu: <4>[753138.683360] CPU 0: hi: 0, btch: 1 usd: 0 <4>[753138.689033] CPU 1: hi: 0, btch: 1 usd: 0 <4>[753138.694707] CPU 2: hi: 0, btch: 1 usd: 0 <4>[753138.700379] CPU 3: hi: 0, btch: 1 usd: 0 <4>[753138.706056] CPU 4: hi: 0, btch: 1 usd: 0 <4>[753138.711729] CPU 5: hi: 0, btch: 1 usd: 0 <4>[753138.717402] CPU 6: hi: 0, btch: 1 usd: 0 <4>[753138.723077] CPU 7: hi: 0, btch: 1 usd: 0 <4>[753138.728748] Node 0 DMA32 per-cpu: <4>[753138.732783] CPU 0: hi: 186, btch: 31 usd: 0 <4>[753138.738457] CPU 1: hi: 186, btch: 31 usd: 0 <4>[753138.744156] CPU 2: hi: 186, btch: 31 usd: 0 <4>[753138.749833] CPU 3: hi: 186, btch: 31 usd: 0 <4>[753138.755506] CPU 4: hi: 186, btch: 31 usd: 0 <4>[753138.761182] CPU 5: hi: 186, btch: 31 usd: 0 <4>[753138.766858] CPU 6: hi: 186, btch: 31 usd: 0 <4>[753138.772628] CPU 7: hi: 186, btch: 31 usd: 0 <4>[753138.778302] Node 0 Normal per-cpu: <4>[753138.782434] CPU 0: hi: 186, btch: 31 usd: 149 <4>[753138.788105] CPU 1: hi: 186, btch: 31 usd: 175 <4>[753138.793778] CPU 2: hi: 186, btch: 31 usd: 75 <4>[753138.799451] CPU 3: hi: 186, btch: 31 usd: 46 <4>[753138.805125] CPU 4: hi: 186, btch: 31 usd: 111 <4>[753138.810796] CPU 5: hi: 186, btch: 31 usd: 159 <4>[753138.816469] CPU 6: hi: 186, btch: 31 usd: 46 <4>[753138.822144] CPU 7: hi: 186, btch: 31 usd: 156 <4>[753138.827819] active_anon:1247111 inactive_anon:79396 isolated_anon:0 <4>[753138.827821] active_file:13315963 inactive_file:13316159 isolated_file:0 <4>[753138.827821] unevictable:8664 dirty:20801 writeback:0 unstable:0 <4>[753138.827821] free:589700 slab_reclaimable:25443478 slab_unreclaimable:11181912 <4>[753138.827822] mapped:11213 shmem:132246 pagetables:3825 bounce:0 <4>[753138.865533] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[753138.907475] lowmem_reserve[]: 0 1880 258420 258420 <4>[753138.913213] Node 0 DMA32 free:387068kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[753138.956232] lowmem_reserve[]: 0 0 256540 256540 <4>[753138.961686] Node 0 Normal free:1953388kB min:67088kB low:83860kB high:100632kB active_anon:4990396kB inactive_anon:317584kB active_file:53264008kB inactive_file:53264244kB unevictable:34656kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:83204kB writeback:0kB mapped:44784kB shmem:528984kB slab_reclaimable:101773740kB slab_unreclaimable:44728084kB kernel_stack:75152kB pagetables:15352kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[753139.012218] lowmem_reserve[]: 0 0 0 0 <4>[753139.016704] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[753139.029264] Node 0 DMA32: 7*4kB 6*8kB 5*16kB 9*32kB 7*64kB 5*128kB 6*256kB 10*512kB 8*1024kB 5*2048kB 88*4096kB = 387068kB <4>[753139.042318] Node 0 Normal: 481723*4kB 1366*8kB 441*16kB 98*32kB 32*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1950060kB <4>[753139.056536] 26765411 total pagecache pages <4>[753139.061434] 0 pages in swap cache <4>[753139.065446] Swap cache stats: add 0, delete 0, find 0/0 <4>[753139.071603] Free swap = 0kB <4>[753139.075136] Total swap = 0kB <6>[753139.474564] 67108863 pages RAM <6>[753139.478294] 1015827 pages reserved <6>[753139.482408] 23510636 pages shared <6>[753139.486427] 41731147 pages non-shared <3>[753139.491054] LustreError: 14453:0:(lprocfs_status.c:1045:lprocfs_stats_alloc_one()) LNET: out of memory at /tmp/rpmbuild-lustre-jsimmons-mRppNlWn/BUILD/lustre-2.8.0/lustre/obdclass/lprocfs_status.c:1045 (tried to alloc '(stats->ls_percpu[cpuid])' = 4352) <3>[753139.517114] LustreError: 14453:0:(lprocfs_status.c:1045:lprocfs_stats_alloc_one()) LNET: 1316767656 total bytes allocated by lnet <4>[753282.255594] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[753282.267727] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[753384.057025] ptlrpcd_01_00: page allocation failure. order:1, mode:0x20 <4>[753384.064646] Pid: 14923, comm: ptlrpcd_01_00 Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[753384.074441] Call Trace: <4>[753384.077505] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[753384.085124] [] ? kmem_getpages+0x62/0x170 <4>[753384.091769] [] ? fallback_alloc+0x1ba/0x270 <4>[753384.098608] [] ? cache_grow+0x2cf/0x320 <4>[753384.105066] [] ? ____cache_alloc_node+0x99/0x160 <4>[753384.112410] [] ? lprocfs_stats_alloc_one+0x84/0x360 [obdclass] <4>[753384.121311] [] ? __kmalloc+0x199/0x230 <4>[753384.127692] [] ? lprocfs_stats_alloc_one+0x84/0x360 [obdclass] <4>[753384.136623] [] ? lprocfs_counter_add+0x1a8/0x1c0 [obdclass] <4>[753384.145358] [] ? ptl_send_rpc+0x5be/0xea0 [ptlrpc] <4>[753384.152916] [] ? ptlrpc_send_new_req+0x50e/0x9b0 [ptlrpc] <4>[753384.161117] [] ? schedule+0x3ee/0xb70 <4>[753384.167401] [] ? ptlrpc_check_set+0x978/0x1d80 [ptlrpc] <4>[753384.175417] [] ? try_to_del_timer_sync+0x7b/0xe0 <4>[753384.182783] [] ? ptlrpcd_check+0x533/0x550 [ptlrpc] <4>[753384.190416] [] ? ptlrpcd+0x27a/0x500 [ptlrpc] <4>[753384.197472] [] ? default_wake_function+0x0/0x20 <4>[753384.204761] [] ? ptlrpcd+0x0/0x500 [ptlrpc] <4>[753384.211600] [] ? kthread+0x9e/0xc0 <4>[753384.217587] [] ? child_rip+0xa/0x20 <4>[753384.223697] [] ? kthread+0x0/0xc0 <4>[753384.229563] [] ? child_rip+0x0/0x20 <6>[753384.235646] Mem-Info: <4>[753384.238497] Node 0 DMA per-cpu: <4>[753384.242367] CPU 0: hi: 0, btch: 1 usd: 0 <4>[753384.248049] CPU 1: hi: 0, btch: 1 usd: 0 <4>[753384.253741] CPU 2: hi: 0, btch: 1 usd: 0 <4>[753384.259435] CPU 3: hi: 0, btch: 1 usd: 0 <4>[753384.265116] CPU 4: hi: 0, btch: 1 usd: 0 <4>[753384.270798] CPU 5: hi: 0, btch: 1 usd: 0 <4>[753384.276521] CPU 6: hi: 0, btch: 1 usd: 0 <4>[753384.282202] CPU 7: hi: 0, btch: 1 usd: 0 <4>[753384.287872] Node 0 DMA32 per-cpu: <4>[753384.291919] CPU 0: hi: 186, btch: 31 usd: 0 <4>[753384.297592] CPU 1: hi: 186, btch: 31 usd: 0 <4>[753384.303285] CPU 2: hi: 186, btch: 31 usd: 0 <4>[753384.308955] CPU 3: hi: 186, btch: 31 usd: 0 <4>[753384.314631] CPU 4: hi: 186, btch: 31 usd: 0 <4>[753384.320351] CPU 5: hi: 186, btch: 31 usd: 0 <4>[753384.326023] CPU 6: hi: 186, btch: 31 usd: 0 <4>[753384.331693] CPU 7: hi: 186, btch: 31 usd: 0 <4>[753384.337363] Node 0 Normal per-cpu: <4>[753384.341493] CPU 0: hi: 186, btch: 31 usd: 175 <4>[753384.347196] CPU 1: hi: 186, btch: 31 usd: 7 <4>[753384.352868] CPU 2: hi: 186, btch: 31 usd: 10 <4>[753384.358578] CPU 3: hi: 186, btch: 31 usd: 9 <4>[753384.364253] CPU 4: hi: 186, btch: 31 usd: 67 <4>[753384.369920] CPU 5: hi: 186, btch: 31 usd: 183 <4>[753384.375594] CPU 6: hi: 186, btch: 31 usd: 13 <4>[753384.381340] CPU 7: hi: 186, btch: 31 usd: 11 <4>[753384.387004] active_anon:2324248 inactive_anon:87230 isolated_anon:0 <4>[753384.387004] active_file:12988263 inactive_file:12988530 isolated_file:0 <4>[753384.387004] unevictable:8664 dirty:48817 writeback:0 unstable:0 <4>[753384.387005] free:265962 slab_reclaimable:25441586 slab_unreclaimable:11076406 <4>[753384.387005] mapped:11207 shmem:132198 pagetables:5891 bounce:0 <4>[753384.430345] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[753384.472337] lowmem_reserve[]: 0 1880 258420 258420 <4>[753384.478098] Node 0 DMA32 free:387068kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[753384.521109] lowmem_reserve[]: 0 0 256540 256540 <4>[753384.526565] Node 0 Normal free:661040kB min:67088kB low:83860kB high:100632kB active_anon:9296992kB inactive_anon:348920kB active_file:51953052kB inactive_file:51954120kB unevictable:34656kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:195268kB writeback:0kB mapped:44828kB shmem:528792kB slab_reclaimable:101766344kB slab_unreclaimable:44305624kB kernel_stack:75056kB pagetables:23564kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[753384.577138] lowmem_reserve[]: 0 0 0 0 <4>[753384.581620] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[753384.594185] Node 0 DMA32: 7*4kB 6*8kB 5*16kB 9*32kB 7*64kB 5*128kB 6*256kB 10*512kB 8*1024kB 5*2048kB 88*4096kB = 387068kB <4>[753384.607227] Node 0 Normal: 159789*4kB 1411*8kB 427*16kB 68*32kB 19*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 660668kB <4>[753384.621339] 26110047 total pagecache pages <4>[753384.626229] 0 pages in swap cache <4>[753384.630249] Swap cache stats: add 0, delete 0, find 0/0 <4>[753384.636411] Free swap = 0kB <4>[753384.639955] Total swap = 0kB <6>[753385.018473] 67108863 pages RAM <6>[753385.022215] 1015827 pages reserved <6>[753385.026330] 23251836 pages shared <6>[753385.030350] 42306611 pages non-shared <3>[753385.034828] LustreError: 14923:0:(lprocfs_status.c:1045:lprocfs_stats_alloc_one()) LNET: out of memory at /tmp/rpmbuild-lustre-jsimmons-mRppNlWn/BUILD/lustre-2.8.0/lustre/obdclass/lprocfs_status.c:1045 (tried to alloc '(stats->ls_percpu[cpuid])' = 4352) <3>[753385.060875] LustreError: 14923:0:(lprocfs_status.c:1045:lprocfs_stats_alloc_one()) LNET: 1329830920 total bytes allocated by lnet <4>[753699.349124] ptlrpcd_01_02: page allocation failure. order:1, mode:0x20 <4>[753699.356745] Pid: 14925, comm: ptlrpcd_01_02 Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[753699.366530] Call Trace: <4>[753699.369577] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[753699.378114] [] ? select_idle_sibling+0x95/0x150 <4>[753699.385349] [] ? kmem_getpages+0x62/0x170 <4>[753699.392001] [] ? fallback_alloc+0x1ba/0x270 <4>[753699.398845] [] ? cache_grow+0x2cf/0x320 <4>[753699.405299] [] ? ____cache_alloc_node+0x99/0x160 <4>[753699.412628] [] ? kmem_cache_alloc_node_trace+0x90/0x200 <4>[753699.420641] [] ? __kmalloc_node+0x4d/0x60 <4>[753699.427289] [] ? __alloc_skb+0x7a/0x190 <4>[753699.433740] [] ? dev_alloc_skb+0x1d/0x40 <4>[753699.440298] [] ? ipoib_alloc_rx_skb+0x3f/0x200 [ib_ipoib] <4>[753699.448507] [] ? mlx4_ib_poll_cq+0x52a/0xd30 [mlx4_ib] <4>[753699.456422] [] ? ipoib_ib_handle_rx_wc+0x8c/0x300 [ib_ipoib] <4>[753699.465149] [] ? ipoib_poll+0x14b/0x180 [ib_ipoib] <4>[753699.472675] [] ? net_rx_action+0x103/0x300 <4>[753699.479421] [] ? __do_softirq+0xe5/0x230 <4>[753699.485986] [] ? mlx4_msi_x_interrupt+0x14/0x20 [mlx4_core] <4>[753699.494599] [] ? call_softirq+0x1c/0x30 <4>[753699.501047] [] ? do_softirq+0x65/0xa0 <4>[753699.507302] [] ? irq_exit+0x85/0x90 <4>[753699.513366] [] ? do_IRQ+0x75/0xf0 <4>[753699.519233] [] ? ret_from_intr+0x0/0x11 <4>[753699.525684] [] ? clear_page_dirty_for_io+0x3f/0xf0 <4>[753699.533918] [] ? vvp_page_make_ready+0x3f/0x250 [lustre] <4>[753699.542046] [] ? cl_page_make_ready+0x89/0x350 [obdclass] <4>[753699.550254] [] ? osc_extent_make_ready+0x3ad/0xe40 [osc] <4>[753699.558368] [] ? lnet_prep_send+0x50/0xb0 [lnet] <4>[753699.565697] [] ? __wake_up+0x53/0x70 <4>[753699.571933] [] ? osc_io_unplug0+0x10ce/0x1b10 [osc] <4>[753699.579551] [] ? LNetMDAttach+0x54f/0x720 [lnet] <4>[753699.586874] [] ? __dequeue_entity+0x30/0x50 <4>[753699.593715] [] ? sys_execve+0x60/0x80 <4>[753699.600007] [] ? ptlrpc_unregister_reply+0x6c/0x810 [ptlrpc] <4>[753699.608722] [] ? osc_io_unplug+0x10/0x20 [osc] <4>[753699.615854] [] ? brw_queue_work+0x3b/0xf0 [osc] <4>[753699.623119] [] ? work_interpreter+0x30/0x100 [ptlrpc] <4>[753699.630952] [] ? ptlrpc_check_set+0x615/0x1d80 [ptlrpc] <4>[753699.638967] [] ? try_to_del_timer_sync+0x7b/0xe0 <4>[753699.646330] [] ? ptlrpcd_check+0x533/0x550 [ptlrpc] <4>[753699.653976] [] ? ptlrpcd+0x31b/0x500 [ptlrpc] <4>[753699.661012] [] ? default_wake_function+0x0/0x20 <4>[753699.668275] [] ? ptlrpcd+0x0/0x500 [ptlrpc] <4>[753699.675115] [] ? kthread+0x9e/0xc0 <4>[753699.681081] [] ? child_rip+0xa/0x20 <4>[753699.687142] [] ? kthread+0x0/0xc0 <4>[753699.693014] [] ? child_rip+0x0/0x20 <4>[753767.468319] Lustre: atlas1-MDT0000: haven't heard from client df8da4eb-dc3b-62ec-f8bc-c2bc7f5c7cd7 (at 10059@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f1008d400, cur 1487248869 expire 1487247969 last 1487247517 <4>[753767.493645] Lustre: Skipped 1 previous similar message <4>[754102.329209] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[754102.341340] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[754654.711592] swapper: page allocation failure. order:1, mode:0x20 <4>[754654.718628] Pid: 0, comm: swapper Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[754654.727424] Call Trace: <4>[754654.730473] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[754654.739005] [] ? kmem_getpages+0x62/0x170 <4>[754654.745653] [] ? fallback_alloc+0x1ba/0x270 <4>[754654.752494] [] ? cache_grow+0x2cf/0x320 <4>[754654.758948] [] ? ____cache_alloc_node+0x99/0x160 <4>[754654.766276] [] ? kmem_cache_alloc_node_trace+0x90/0x200 <4>[754654.774285] [] ? __kmalloc_node+0x4d/0x60 <4>[754654.780934] [] ? __alloc_skb+0x7a/0x190 <4>[754654.787385] [] ? dev_alloc_skb+0x1d/0x40 <4>[754654.793940] [] ? ipoib_alloc_rx_skb+0x3f/0x200 [ib_ipoib] <4>[754654.802144] [] ? mlx4_ib_poll_cq+0x52a/0xd30 [mlx4_ib] <4>[754654.810055] [] ? ipoib_ib_handle_rx_wc+0x8c/0x300 [ib_ipoib] <4>[754654.818764] [] ? ipoib_poll+0x14b/0x180 [ib_ipoib] <4>[754654.826284] [] ? net_rx_action+0x103/0x300 <4>[754654.833028] [] ? __do_softirq+0xe5/0x230 <4>[754654.839587] [] ? mlx4_msi_x_interrupt+0x14/0x20 [mlx4_core] <4>[754654.848198] [] ? call_softirq+0x1c/0x30 <4>[754654.854648] [] ? do_softirq+0x65/0xa0 <4>[754654.860906] [] ? irq_exit+0x85/0x90 <4>[754654.866970] [] ? do_IRQ+0x75/0xf0 <4>[754654.872836] [] ? ret_from_intr+0x0/0x11 <4>[754654.879288] [] ? intel_idle+0xfe/0x1b0 <4>[754654.886341] [] ? intel_idle+0xe1/0x1b0 <4>[754654.892701] [] ? sched_clock+0x9/0x10 <4>[754654.898955] [] ? sched_clock_cpu+0xcd/0x110 <4>[754654.905789] [] ? cpuidle_idle_call+0x7a/0xe0 <4>[754654.912725] [] ? cpu_idle+0xb6/0x110 <4>[754654.918885] [] ? start_secondary+0x2c0/0x316 <4>[754864.762929] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[754864.775064] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[755628.162499] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[755631.261480] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[755633.339075] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[755901.945899] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[755901.958026] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[755923.128231] kiblnd_sd_01_01: page allocation failure. order:1, mode:0x20 <4>[755923.136134] Pid: 14830, comm: kiblnd_sd_01_01 Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[755923.146109] Call Trace: <4>[755923.149158] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[755923.157700] [] ? kmem_getpages+0x62/0x170 <4>[755923.164345] [] ? fallback_alloc+0x1ba/0x270 <4>[755923.171187] [] ? cache_grow+0x2cf/0x320 <4>[755923.177700] [] ? ____cache_alloc_node+0x99/0x160 <4>[755923.185027] [] ? kmem_cache_alloc_node_trace+0x90/0x200 <4>[755923.193031] [] ? __kmalloc_node+0x4d/0x60 <4>[755923.199682] [] ? __alloc_skb+0x7a/0x190 <4>[755923.206175] [] ? dev_alloc_skb+0x1d/0x40 <4>[755923.212730] [] ? ipoib_alloc_rx_skb+0x3f/0x200 [ib_ipoib] <4>[755923.220935] [] ? mlx4_ib_poll_cq+0x52a/0xd30 [mlx4_ib] <4>[755923.228848] [] ? ipoib_ib_handle_rx_wc+0x8c/0x300 [ib_ipoib] <4>[755923.237551] [] ? ipoib_poll+0x14b/0x180 [ib_ipoib] <4>[755923.245075] [] ? net_rx_action+0x103/0x300 <4>[755923.251820] [] ? __do_softirq+0xe5/0x230 <4>[755923.258380] [] ? mlx4_msi_x_interrupt+0x14/0x20 [mlx4_core] <4>[755923.266989] [] ? call_softirq+0x1c/0x30 <4>[755923.273441] [] ? do_softirq+0x65/0xa0 <4>[755923.284830] [] ? irq_exit+0x85/0x90 <4>[755923.290891] [] ? do_IRQ+0x75/0xf0 <4>[755923.296760] [] ? ret_from_intr+0x0/0x11 <4>[755923.303210] [] ? kiblnd_scheduler+0x412/0x1160 [ko2iblnd] <4>[755923.312319] [] ? default_wake_function+0x0/0x20 <4>[755923.319553] [] ? kiblnd_scheduler+0x0/0x1160 [ko2iblnd] <4>[755923.327571] [] ? kthread+0x9e/0xc0 <4>[755923.333632] [] ? child_rip+0xa/0x20 <4>[755923.339697] [] ? kthread+0x0/0xc0 <4>[755923.345564] [] ? child_rip+0x0/0x20 <4>[756020.701356] swapper: page allocation failure. order:1, mode:0x20 <4>[756020.708390] Pid: 0, comm: swapper Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[756020.717192] Call Trace: <4>[756020.720240] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[756020.728774] [] ? kmem_getpages+0x62/0x170 <4>[756020.735420] [] ? fallback_alloc+0x1ba/0x270 <4>[756020.742361] [] ? cache_grow+0x2cf/0x320 <4>[756020.748813] [] ? ____cache_alloc_node+0x99/0x160 <4>[756020.756140] [] ? kmem_cache_alloc_node_trace+0x90/0x200 <4>[756020.764146] [] ? __kmalloc_node+0x4d/0x60 <4>[756020.770800] [] ? __alloc_skb+0x7a/0x190 <4>[756020.777250] [] ? dev_alloc_skb+0x1d/0x40 <4>[756020.783804] [] ? ipoib_alloc_rx_skb+0x3f/0x200 [ib_ipoib] <4>[756020.792007] [] ? mlx4_ib_poll_cq+0x52a/0xd30 [mlx4_ib] <4>[756020.799932] [] ? ipoib_ib_handle_rx_wc+0x8c/0x300 [ib_ipoib] <4>[756020.808650] [] ? ipoib_poll+0x14b/0x180 [ib_ipoib] <4>[756020.816163] [] ? net_rx_action+0x103/0x300 <4>[756020.822906] [] ? __do_softirq+0xe5/0x230 <4>[756020.829488] [] ? mlx4_msi_x_interrupt+0x14/0x20 [mlx4_core] <4>[756020.838106] [] ? call_softirq+0x1c/0x30 <4>[756020.844557] [] ? do_softirq+0x65/0xa0 <4>[756020.850815] [] ? irq_exit+0x85/0x90 <4>[756020.856886] [] ? do_IRQ+0x75/0xf0 <4>[756020.862776] [] ? ret_from_intr+0x0/0x11 <4>[756020.869233] [] ? intel_idle+0xfe/0x1b0 <4>[756020.876287] [] ? intel_idle+0xe1/0x1b0 <4>[756020.882640] [] ? cpuidle_idle_call+0x7a/0xe0 <4>[756020.889575] [] ? cpu_idle+0xb6/0x110 <4>[756020.895734] [] ? start_secondary+0x2c0/0x316 <4>[756510.895667] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[756510.907795] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[757206.458423] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[757206.470550] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[758052.179984] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[758052.192110] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[758532.805715] Lustre: Unmounted atlas1-client <4>[758668.453562] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[758668.465687] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[759776.526636] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[759776.538763] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[760727.757844] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[760727.769972] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[761403.913942] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[761403.926070] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[762291.790541] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[762291.802666] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[762933.230442] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[762933.242576] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[763766.181619] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[763766.193742] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[764441.513809] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[764441.525953] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[765094.157754] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[765094.169881] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[765765.773837] LNet: No route to 12345-10.38.144.10@o2ib5 via 10.36.226.117@o2ib200 (all routers down) <4>[765765.784510] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1487260867/real 1487260867] req@ffff88283d968080 x1558709204413456/t0(0) o104->atlas1-MDT0000@10.38.144.10@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1487261434 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[765766.274535] LNet: No route to 12345-10.38.144.10@o2ib5 via 10.36.226.117@o2ib200 (all routers down) <4>[765766.285231] LNet: Skipped 83352 previous similar messages <4>[765766.291602] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1487260867/real 1487260867] req@ffff88283d968080 x1558709204413456/t0(0) o104->atlas1-MDT0000@10.38.144.10@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1487261434 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 <4>[765766.324089] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 83352 previous similar messages <4>[765767.276507] LNet: No route to 12345-10.38.144.10@o2ib5 via 10.36.226.117@o2ib200 (all routers down) <4>[765767.287194] LNet: Skipped 171643 previous similar messages <4>[765767.293667] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1487260868/real 1487260868] req@ffff88283d968080 x1558709204413456/t0(0) o104->atlas1-MDT0000@10.38.144.10@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1487261435 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 <4>[765767.326255] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 171643 previous similar messages <4>[765769.278503] LNet: No route to 12345-10.38.144.10@o2ib5 via 10.36.226.117@o2ib200 (all routers down) <4>[765769.289185] LNet: Skipped 355862 previous similar messages <4>[765769.295653] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1487260870/real 1487260870] req@ffff88283d968080 x1558709204413456/t0(0) o104->atlas1-MDT0000@10.38.144.10@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1487261437 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 <4>[765769.328203] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 355862 previous similar messages <4>[765773.280500] LNet: No route to 12345-10.38.144.10@o2ib5 via 10.36.226.117@o2ib200 (all routers down) <4>[765773.291177] LNet: Skipped 726350 previous similar messages <4>[765773.297662] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1487260874/real 1487260874] req@ffff88283d968080 x1558709204413456/t0(0) o104->atlas1-MDT0000@10.38.144.10@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1487261441 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 <4>[765773.330196] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 726350 previous similar messages <4>[765781.282486] LNet: No route to 12345-10.38.144.10@o2ib5 via 10.36.226.117@o2ib200 (all routers down) <4>[765781.293176] LNet: Skipped 1462118 previous similar messages <4>[765781.299743] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1487260882/real 1487260882] req@ffff88283d968080 x1558709204413456/t0(0) o104->atlas1-MDT0000@10.38.144.10@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1487261449 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 <4>[765781.332283] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 1462118 previous similar messages <4>[765797.284466] LNet: No route to 12345-10.38.144.10@o2ib5 via 10.36.226.117@o2ib200 (all routers down) <4>[765797.295162] LNet: Skipped 2927154 previous similar messages <4>[765797.301709] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1487260898/real 1487260898] req@ffff88283d968080 x1558709204413456/t0(0) o104->atlas1-MDT0000@10.38.144.10@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1487261465 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 <4>[765797.334251] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 2927154 previous similar messages <4>[765829.286402] LNet: No route to 12345-10.38.144.10@o2ib5 via 10.36.226.117@o2ib200 (all routers down) <4>[765829.297920] LNet: Skipped 5910233 previous similar messages <4>[765829.305202] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1487260930/real 1487260930] req@ffff88283d968080 x1558709204413456/t0(0) o104->atlas1-MDT0000@10.38.144.10@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1487261497 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 <4>[765829.338486] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 5910233 previous similar messages <4>[765878.452026] Lustre: atlas1-MDT0000: haven't heard from client 3b2537dd-163b-7f9d-fde1-1ee326e14461 (at 7936@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0e39d400, cur 1487260980 expire 1487260080 last 1487259628 <4>[765878.477203] Lustre: Skipped 250 previous similar messages <4>[765893.288314] LNet: No route to 12345-10.38.144.10@o2ib5 via 10.36.226.117@o2ib200 (all routers down) <4>[765893.298989] LNet: Skipped 11823057 previous similar messages <4>[765893.306314] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1487260994/real 1487260994] req@ffff88283d968080 x1558709204413456/t0(0) o104->atlas1-MDT0000@10.38.144.10@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1487261561 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 <4>[765893.343991] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 11823178 previous similar messages <4>[765947.077505] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[765947.089632] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[766021.290152] LNet: No route to 12345-10.38.144.10@o2ib5 via 10.36.226.117@o2ib200 (all routers down) <4>[766021.300869] LNet: Skipped 23536569 previous similar messages <4>[766021.308150] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1487261122/real 1487261122] req@ffff88283d968080 x1558709204413456/t0(0) o104->atlas1-MDT0000@10.38.144.10@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1487261689 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 <4>[766021.340786] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 23536562 previous similar messages <4>[766277.291791] LNet: No route to 12345-10.38.144.10@o2ib5 via 10.36.226.117@o2ib200 (all routers down) <4>[766277.302505] LNet: Skipped 47358799 previous similar messages <4>[766277.309793] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1487261378/real 1487261378] req@ffff88283d968080 x1558709204413456/t0(0) o104->atlas1-MDT0000@10.38.144.10@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1487261945 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 <4>[766277.342283] Lustre: 15809:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 47358799 previous similar messages <4>[766329.451111] Lustre: atlas1-MDT0000: haven't heard from client 8a3e9aed-4c4e-7325-145e-34df455db76c (at 10.38.144.111@o2ib5) in 980 seconds. I think it's dead, and I am evicting it. exp ffff8807e0980800, cur 1487261431 expire 1487260531 last 1487260451 <4>[766360.772715] Lustre: 15874:0:(service.c:1336:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[766360.772716] req@ffff8828381ba080 x1549207904390360/t0(0) o101->ab4e429d-1b8c-bbdb-2826-67af15b260c0@10.36.205.206@o2ib:557/0 lens 4840/3512 e 12 to 0 dl 1487261467 ref 2 fl Interpret:/0/0 rc 0/0 <4>[766541.917846] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[766541.929970] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[766615.772471] Lustre: atlas1-MDT0000: Client ab4e429d-1b8c-bbdb-2826-67af15b260c0 (at 10.36.205.206@o2ib) reconnecting <6>[766615.784795] Lustre: atlas1-MDT0000: Connection restored to ab4e429d-1b8c-bbdb-2826-67af15b260c0 (at 10.36.205.206@o2ib) <3>[766665.773279] LustreError: 15809:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) ### client (nid 10.38.144.10@o2ib5) failed to reply to blocking AST (req status 0 rc -110), evict it ns: mdt-atlas1-MDT0000_UUID lock: ffff881f826975c0/0xd94b6744897f6e18 lrc: 4/0,0 mode: PR/PR res: [0x200384ee0:0x1a196:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.38.144.10@o2ib5 remote: 0x5e534647adbffb85 expref: 489 pid: 15994 timeout: 5062232925 lvb_type: 0 <3>[766665.820003] LustreError: 138-a: atlas1-MDT0000: A client on nid 10.38.144.10@o2ib5 was evicted due to a lock blocking callback time out: rc -110 <4>[766665.847186] Lustre: 15809:0:(service.c:2097:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:300s); client may timeout. req@ffff8828381ba080 x1549207904390360/t647759125747(0) o101->ab4e429d-1b8c-bbdb-2826-67af15b260c0@10.36.205.206@o2ib:557/0 lens 4840/672 e 12 to 0 dl 1487261467 ref 1 fl Complete:/0/0 rc 0/0 <4>[766780.452050] Lustre: atlas1-MDT0000: haven't heard from client 3a925634-28e6-0ea6-737d-d7150925cf0e (at 10.38.145.172@o2ib5) in 1351 seconds. I think it's dead, and I am evicting it. exp ffff883ecfb72000, cur 1487261882 expire 1487260982 last 1487260531 <4>[766780.478023] Lustre: Skipped 15 previous similar messages <6>[767361.922165] Lustre: atlas1-MDT0000: Connection restored to 206a5ca8-dd4a-decd-093a-cfea43218a0f (at 10.36.207.152@o2ib) <4>[767710.191252] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[767710.203379] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[768043.813056] Lustre: atlas1-MDT0000: Connection restored to 3222a972-a17a-24bf-93e1-701b3ec76dc5 (at 10.38.144.69@o2ib5) <6>[768046.003117] Lustre: atlas1-MDT0000: Connection restored to 97a44fed-b26f-f97a-9a4e-bca2701b75b7 (at 10.38.144.211@o2ib5) <6>[768046.015820] Lustre: Skipped 3 previous similar messages <6>[768051.760734] Lustre: atlas1-MDT0000: Connection restored to d944d3b1-af8b-2e7b-5dc1-043f6d313467 (at 10.38.145.239@o2ib5) <6>[768051.773442] Lustre: Skipped 3 previous similar messages <6>[768059.843321] Lustre: atlas1-MDT0000: Connection restored to d4af4dd2-e98d-1b8f-51d9-4945e94443e5 (at 10.38.144.96@o2ib5) <6>[768059.855924] Lustre: Skipped 10 previous similar messages <6>[768076.794005] Lustre: atlas1-MDT0000: Connection restored to c5310649-35d2-c54f-19af-76c39114b82f (at 10.38.144.22@o2ib5) <6>[768076.806610] Lustre: Skipped 22 previous similar messages <6>[768114.925921] Lustre: atlas1-MDT0000: Connection restored to ec6c7af3-2e4f-9e81-ca3d-7f4a2d9eb8c5 (at 10.38.144.151@o2ib5) <6>[768114.938629] Lustre: Skipped 256 previous similar messages <4>[768315.820460] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[768315.832601] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[768315.845024] Lustre: Skipped 178 previous similar messages <4>[768780.778003] Lustre: atlas1-MDT0000: Client 97a44fed-b26f-f97a-9a4e-bca2701b75b7 (at 10.38.144.211@o2ib5) reconnecting <6>[768780.790524] Lustre: atlas1-MDT0000: Connection restored to 97a44fed-b26f-f97a-9a4e-bca2701b75b7 (at 10.38.144.211@o2ib5) <6>[768780.803227] Lustre: Skipped 4 previous similar messages <4>[770021.301229] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[770024.397799] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[770026.527634] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[771096.444248] Lustre: atlas1-MDT0000: haven't heard from client b486839b-3c52-e14d-25bd-b187f2bb87ed (at 10.38.144.242@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880cb8d9dc00, cur 1487266198 expire 1487265298 last 1487264846 <4>[771096.470299] Lustre: Skipped 563 previous similar messages <4>[772313.442549] Lustre: atlas1-MDT0000: haven't heard from client 206a5ca8-dd4a-decd-093a-cfea43218a0f (at 10.36.207.152@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f12e71000, cur 1487267415 expire 1487266515 last 1487266063 <4>[772313.468447] Lustre: Skipped 481 previous similar messages <6>[773636.324862] Lustre: atlas1-MDT0000: Connection restored to 206a5ca8-dd4a-decd-093a-cfea43218a0f (at 10.36.207.152@o2ib) <6>[774449.588958] Lustre: atlas1-MDT0000: Connection restored to df5a8e60-e3d6-c2a9-b9e2-49734c634cde (at 10.38.146.49@o2ib5) <6>[774454.392476] Lustre: atlas1-MDT0000: Connection restored to 90601d4e-b8f5-3ad5-a331-11294281ae6b (at 10.38.146.51@o2ib5) <6>[774465.264795] Lustre: atlas1-MDT0000: Connection restored to df66000a-b89d-d830-7b57-be9a8c411558 (at 10.38.146.50@o2ib5) <6>[774465.277400] Lustre: Skipped 6 previous similar messages <6>[774978.071943] Lustre: atlas1-MDT0000: Connection restored to 55e74c02-8c85-24c3-3ca9-3f8c442013bf (at 10.38.144.20@o2ib5) <6>[775229.759028] Lustre: atlas1-MDT0000: Connection restored to 6b1dbd69-68f0-4dfa-c0fb-a26c54bdcf5c (at 10.38.144.85@o2ib5) <6>[775294.698728] Lustre: atlas1-MDT0000: Connection restored to a0fb56f1-05e2-bf7f-1b99-0ccc8fba3f7b (at 10.38.144.75@o2ib5) <6>[775294.711323] Lustre: Skipped 64 previous similar messages <6>[775574.782469] Lustre: atlas1-MDT0000: Connection restored to d943161c-706c-9c8e-9f14-c8d2a2bd1980 (at 10.38.144.215@o2ib5) <6>[775574.795170] Lustre: Skipped 38 previous similar messages <6>[775831.612304] Lustre: atlas1-MDT0000: Connection restored to 5f5b1fe9-58c8-e5f5-e912-fe71d0512347 (at 10.38.145.75@o2ib5) <6>[775831.624902] Lustre: Skipped 161 previous similar messages <4>[776304.946701] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[776691.770404] Lustre: atlas1-MDT0000: Connection restored to 34d5a529-36d7-ec32-ab74-6e7c7b7a17bd (at 10.38.144.11@o2ib5) <6>[776691.783022] Lustre: Skipped 247 previous similar messages <4>[777332.012025] Lustre: atlas1-MDT0000: Client 835099db-203b-44fa-2e02-0c72784e505c (at 9298@gni100) reconnecting <6>[777332.029768] Lustre: atlas1-MDT0000: Connection restored to 835099db-203b-44fa-2e02-0c72784e505c (at 9298@gni100) <6>[778016.018217] Lustre: atlas1-MDT0000: Connection restored to 9788e3cd-e58d-5745-6e2d-7c92ca6de3e7 (at 10.39.232.67@o2ib6) <6>[778016.030841] Lustre: Skipped 1 previous similar message <4>[778883.660276] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[778883.672398] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[778883.684801] Lustre: Skipped 66 previous similar messages <4>[778962.435727] Lustre: atlas1-MDT0000: haven't heard from client 9d66a10d-8161-ab9f-7149-a62be632c968 (at 10.39.232.69@o2ib6) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8827eac60000, cur 1487274064 expire 1487273164 last 1487272712 <4>[779413.433273] Lustre: atlas1-MDT0000: haven't heard from client fa26f657-20f1-d449-b139-56787623ac91 (at 10.36.202.41@o2ib) in 1155 seconds. I think it's dead, and I am evicting it. exp ffff883edc361c00, cur 1487274515 expire 1487273615 last 1487273360 <4>[779413.459093] Lustre: Skipped 10 previous similar messages <6>[779894.622331] Lustre: atlas1-MDT0000: Connection restored to 67c71f2f-f7b7-0993-8b8a-77053c94b4cb (at 10.36.202.41@o2ib) <6>[779894.634835] Lustre: Skipped 3 previous similar messages <4>[779992.825475] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[780818.430421] Lustre: atlas1-MDT0000: haven't heard from client dffa1855-1951-7456-7799-d1c59a7c4245 (at 14354@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0ef47c00, cur 1487275920 expire 1487275020 last 1487274568 <4>[780818.455725] Lustre: Skipped 2 previous similar messages <4>[780909.613434] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[780909.625574] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[780909.637979] Lustre: Skipped 10 previous similar messages <4>[781574.277357] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[781574.289495] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[782180.286804] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[782180.298980] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[782856.380919] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[782856.393080] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[782899.427539] Lustre: atlas1-MDT0000: haven't heard from client 68f6dfd7-9bde-79f1-fab2-5e3d5074ebb1 (at 225@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0edf9c00, cur 1487278001 expire 1487277101 last 1487276649 <4>[783481.791113] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[783481.803279] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[784072.049327] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[784423.008414] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[784425.986416] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[784428.036297] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[784683.300504] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[784683.312685] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[784683.325144] Lustre: Skipped 1 previous similar message <3>[784979.364262] LustreError: 16072:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a0f53:0x10:0x0]: rc = -2 <3>[784979.379024] LustreError: 16072:0:(mdt_handler.c:893:mdt_getattr_internal()) Skipped 2 previous similar messages <4>[785297.845579] Lustre: atlas1-MDT0000: Client a88a9b08-b013-7efa-0361-5630c695077d (at 30@gni2) reconnecting <6>[785297.856889] Lustre: atlas1-MDT0000: Connection restored to a88a9b08-b013-7efa-0361-5630c695077d (at 30@gni2) <4>[785306.946563] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[785940.961467] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[785940.973592] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[785940.986050] Lustre: Skipped 1 previous similar message <4>[786525.713715] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[787152.508699] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[787152.520864] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[787152.533290] Lustre: Skipped 1 previous similar message <4>[787828.240684] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[787828.252818] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[788403.447007] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[788403.459144] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[788994.265182] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[788994.277330] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[789833.666184] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[789833.678333] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <3>[789949.401282] LustreError: 15683:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20039f292:0x16ebb:0x0]: rc = -2 <4>[790412.785282] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[790433.885944] Lustre: atlas1-MDT0000: Connection restored to 67ed7009-81c9-b39c-7e89-536983a994d6 (at 9965@gni100) <6>[790433.897867] Lustre: Skipped 450 previous similar messages <3>[790568.059819] LustreError: 15966:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) ### client (nid 10.38.144.136@o2ib5) failed to reply to blocking AST (req status 0 rc -110), evict it ns: mdt-atlas1-MDT0000_UUID lock: ffff88128ddc4c80/0xd94b67462e504069 lrc: 4/0,0 mode: PR/PR res: [0x2003a1131:0x8d19:0x0].0x0 bits 0x13 rrc: 40 type: IBT flags: 0x60200400000020 nid: 10.38.144.136@o2ib5 remote: 0xde382aea78e28614 expref: 2616 pid: 16240 timeout: 5086135245 lvb_type: 0 <3>[790568.106818] LustreError: 138-a: atlas1-MDT0000: A client on nid 10.38.144.136@o2ib5 was evicted due to a lock blocking callback time out: rc -110 <4>[791012.174696] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[791745.173545] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[791745.185692] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[791745.198151] Lustre: Skipped 1 previous similar message <6>[792304.665807] Lustre: atlas1-MDT0000: Connection restored to 85ba3f77-b9de-d40c-70c9-e14bc6b3e9f4 (at 17390@gni100) <4>[792367.052820] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[792661.926516] Lustre: atlas1-MDT0000: Connection restored to 5cf9d3d4-ee12-4e0a-9380-a2d2e2cead7a (at 7936@gni100) <6>[792661.938470] Lustre: Skipped 60 previous similar messages <4>[793074.224763] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[793074.236898] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[793074.249336] Lustre: Skipped 14 previous similar messages <4>[793678.891871] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[793678.904004] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[794736.774650] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[794736.786841] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[795710.427288] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[795710.439457] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[796496.305032] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[796496.317202] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[797359.006214] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[797359.018358] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[797359.030766] Lustre: Skipped 1 previous similar message <4>[798130.153955] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[798130.166092] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[799599.868231] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[799599.880417] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[800251.050341] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[800251.062496] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[801057.335410] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[801057.347571] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[801656.862475] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[801656.874601] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[801708.083309] Lustre: atlas1-MDT0000: Client 1855f567-a4b5-2fec-a1a4-8a71f15c3219 (at 15935@gni100) reconnecting <6>[801708.095051] Lustre: atlas1-MDT0000: Connection restored to 1855f567-a4b5-2fec-a1a4-8a71f15c3219 (at 15935@gni100) <4>[801710.222475] Lustre: atlas1-MDT0000: Client 824432fa-c3e4-a5e3-f325-4073cd74af8b (at 3726@gni100) reconnecting <4>[801716.037303] Lustre: atlas1-MDT0000: Client 5854076b-d005-bd8d-5954-6b58089ce120 (at 13696@gni100) reconnecting <4>[801716.049032] Lustre: Skipped 3 previous similar messages <4>[802337.972536] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[802337.984653] Lustre: Skipped 5 previous similar messages <6>[802337.990865] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[802338.003303] Lustre: Skipped 10 previous similar messages <4>[802963.402337] Lustre: atlas1-MDT0000: haven't heard from client fde4593d-6e18-90aa-5ab8-0bf848e39cc7 (at 1289@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f09ccb800, cur 1487298065 expire 1487297165 last 1487296713 <4>[802963.427514] Lustre: Skipped 44 previous similar messages <4>[803088.619457] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[803088.631591] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[803732.385681] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[803732.397830] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[804376.367573] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[804376.379743] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[805169.931326] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[805169.943466] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[805818.487318] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[805818.499472] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[806435.654465] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[806435.666628] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[807029.004377] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[807618.039526] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[807618.051666] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[807618.064094] Lustre: Skipped 1 previous similar message <4>[808194.646286] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[808784.872857] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[808784.884989] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[808784.897503] Lustre: Skipped 1 previous similar message <4>[809359.525065] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[809989.517082] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[809989.529228] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[809989.541673] Lustre: Skipped 1 previous similar message <4>[810569.948184] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[811172.126975] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[811172.139131] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[811172.151543] Lustre: Skipped 1 previous similar message <4>[811777.612716] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[811777.624840] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[812529.411808] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[812529.423942] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[813100.955122] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[813781.646125] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[813781.658270] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[813781.670698] Lustre: Skipped 1 previous similar message <4>[814423.839331] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[814423.851492] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[815192.110330] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[815192.122560] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[816474.508603] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[816474.520737] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[817140.563943] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[817140.576109] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[817892.163543] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[817892.175666] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[818532.744619] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[818532.756787] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[819238.731145] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[819238.743282] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[819907.922317] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[819907.934454] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[820511.947551] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[820511.959781] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[821506.377622] Lustre: atlas1-MDT0000: haven't heard from client e75cfab5-1d9f-eb85-c864-bdc19f2b0106 (at 4485@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0c6e9000, cur 1487316608 expire 1487315708 last 1487315256 <4>[821548.657262] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[821548.669400] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[822368.084033] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[822368.096189] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[823458.606058] Lustre: atlas1-MDT0000: Client 419bd99a-628d-7f32-d71c-9363cc2db07b (at 10.36.202.142@o2ib) reconnecting <6>[823458.618390] Lustre: atlas1-MDT0000: Connection restored to 419bd99a-628d-7f32-d71c-9363cc2db07b (at 10.36.202.142@o2ib) <4>[823591.001792] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[823591.013976] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[824426.993518] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[824427.005668] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[825133.298537] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[825133.310673] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[825212.383260] Lustre: atlas1-MDT0000: haven't heard from client fd8234be-2b59-7c2e-851b-7d30198f34cb (at 16201@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f10315c00, cur 1487320314 expire 1487319414 last 1487318962 <4>[826336.711971] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[826336.724117] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[827023.929973] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[827023.942149] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[827605.169003] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[827605.181143] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[828182.290243] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[828182.302411] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[828806.142097] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[828806.154247] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[829444.230248] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[829444.242396] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[830025.538531] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[830025.550762] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[830602.378665] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[830602.390808] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[831202.887107] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[831202.899345] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[832133.002824] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[832133.014963] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[832713.396050] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[832713.408194] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[833293.956689] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[833293.968854] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[833909.690741] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[833909.702863] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[834490.918891] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[834490.931060] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[835070.637199] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[835070.649341] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[835817.428600] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[835817.440769] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[836611.873155] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[836611.885280] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[837821.974608] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[837821.991867] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <3>[837821.992033] LustreError: 42492:0:(ldlm_lib.c:3122:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff883ed16a3050 x1558693977042116/t0(0) o37->49207f95-72eb-913e-ab93-3a579be04014@10.36.225.5@o2ib:605/0 lens 568/440 e 0 to 0 dl 1487333240 ref 1 fl Interpret:/0/0 rc 0/0 <4>[838778.227338] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[838778.239470] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[840097.503553] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[840097.515688] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[840755.377520] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[840755.389677] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[841554.628052] Lustre: atlas1-MDT0000: Client dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) reconnecting <6>[841554.640376] Lustre: atlas1-MDT0000: Connection restored to dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) <4>[841577.767412] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[841577.779562] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[842021.019753] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[842024.040743] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[842026.080000] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[842305.987487] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[842305.999625] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[842970.967751] Lustre: atlas1-MDT0000: Client b7e68d73-10f7-9d2b-ebeb-0f84bb2b2a66 (at 10.36.202.46@o2ib) reconnecting <6>[842970.979988] Lustre: atlas1-MDT0000: Connection restored to b7e68d73-10f7-9d2b-ebeb-0f84bb2b2a66 (at 10.36.202.46@o2ib) <4>[843345.831493] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[843345.843624] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[843963.267741] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[843963.279887] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[844271.046961] Lustre: atlas1-MDT0000: Connection restored to d70690de-d75d-d1d3-c2d4-4f8469e9c3a7 (at 10.39.232.59@o2ib6) <4>[844742.112201] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[844742.124330] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[844838.345083] Lustre: atlas1-MDT0000: haven't heard from client b5fc664d-cf8c-a167-3f17-ee3d062cd1f2 (at 3088@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0f309000, cur 1487339940 expire 1487339040 last 1487338588 <6>[845078.110585] Lustre: atlas1-MDT0000: Connection restored to d5877d20-a2db-ef15-89e3-90b125010650 (at 10.39.232.67@o2ib6) <4>[845356.394435] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[845356.406566] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[846118.341967] Lustre: atlas1-MDT0000: haven't heard from client 9788e3cd-e58d-5745-6e2d-7c92ca6de3e7 (at 10.39.232.67@o2ib6) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f11fefc00, cur 1487341220 expire 1487340320 last 1487339868 <4>[846143.256431] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[846143.268554] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[846481.160372] Lustre: atlas1-MDT0000: Connection restored to 6de1ccb9-0074-a821-fd7d-5f03cf8436ed (at 10.39.232.59@o2ib6) <6>[846603.782230] Lustre: atlas1-MDT0000: Connection restored to 59d55753-7e1c-fd27-f8db-6c5f5b355284 (at 10.39.232.91@o2ib6) <6>[846608.401276] Lustre: atlas1-MDT0000: Connection restored to 1105f478-9598-6d17-59eb-17b15fbcf185 (at 10.39.232.84@o2ib6) <6>[846616.433626] Lustre: atlas1-MDT0000: Connection restored to 33b68e5e-ab22-97b8-ad7c-1277c5726e5f (at 10.39.232.70@o2ib6) <6>[846616.446254] Lustre: Skipped 10 previous similar messages <6>[846758.976385] Lustre: atlas1-MDT0000: Connection restored to 28c9f276-54db-9571-2d2b-a259b0d9fed4 (at 10.39.232.67@o2ib6) <4>[847039.850115] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[847039.862243] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[847550.340383] Lustre: atlas1-MDT0000: haven't heard from client 541c83bc-a3d9-c85b-7357-f2b3831696c7 (at 10.39.232.84@o2ib6) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88110a599c00, cur 1487342652 expire 1487341752 last 1487341300 <4>[848001.339699] Lustre: atlas1-MDT0000: haven't heard from client 7180b0aa-2c6c-89fb-c985-7e3ed55c3247 (at 9208@gni100) in 1008 seconds. I think it's dead, and I am evicting it. exp ffff883f11269000, cur 1487343103 expire 1487342203 last 1487342095 <4>[848001.364898] Lustre: Skipped 13 previous similar messages <4>[848018.522585] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[848018.534704] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[848452.344182] Lustre: atlas1-MDT0000: haven't heard from client 5d34593c-1e49-9e2e-3638-3176d1258910 (at 14348@gni100) in 1002 seconds. I think it's dead, and I am evicting it. exp ffff883f12a85000, cur 1487343554 expire 1487342654 last 1487342552 <4>[848771.210392] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[848771.222520] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[848903.337982] Lustre: atlas1-MDT0000: haven't heard from client f49e6974-0162-9b74-d4a5-20cfef515c0c (at 17196@gni100) in 1350 seconds. I think it's dead, and I am evicting it. exp ffff883f12e59c00, cur 1487344005 expire 1487343105 last 1487342655 <4>[848903.363258] Lustre: Skipped 30 previous similar messages <4>[849354.346481] Lustre: atlas1-MDT0000: haven't heard from client 61378c06-f52e-8884-61d8-8dde0d0d9ba7 (at 8111@gni100) in 1122 seconds. I think it's dead, and I am evicting it. exp ffff883f0cfa2000, cur 1487344456 expire 1487343556 last 1487343334 <4>[849354.371671] Lustre: Skipped 2 previous similar messages <4>[849667.897930] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[849667.910066] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[850930.944760] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[850930.956897] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[851835.894449] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[851835.906608] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[852830.572027] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[852830.584186] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <3>[855865.707428] LustreError: 15935:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a1151:0x87bd:0x0]: rc = -2 <4>[856300.713018] Lustre: atlas1-MDT0000: Client b7e68d73-10f7-9d2b-ebeb-0f84bb2b2a66 (at 10.36.202.46@o2ib) reconnecting <6>[856300.725260] Lustre: atlas1-MDT0000: Connection restored to b7e68d73-10f7-9d2b-ebeb-0f84bb2b2a66 (at 10.36.202.46@o2ib) <4>[856408.647978] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[856408.660130] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[856419.680011] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[856422.797436] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[856424.913270] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <6>[856733.853855] Lustre: atlas1-MDT0000: Connection restored to 07549de6-1334-275c-3c79-482963edf42c (at 10.36.202.19@o2ib) <6>[856733.866368] Lustre: Skipped 2 previous similar messages <4>[857288.048869] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[857288.061025] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[857750.325856] Lustre: atlas1-MDT0000: haven't heard from client 77617551-21d0-9bf4-a8e2-e218ebfcaf39 (at 10.36.202.19@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f16007800, cur 1487352852 expire 1487351952 last 1487351500 <4>[857864.322679] Lustre: atlas1-MDT0000: Client e34a109a-da8b-73a0-b754-8fcb9f358393 (at 10.36.202.35@o2ib) reconnecting <6>[857864.334922] Lustre: atlas1-MDT0000: Connection restored to e34a109a-da8b-73a0-b754-8fcb9f358393 (at 10.36.202.35@o2ib) <4>[858003.779819] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[858003.791967] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[858603.570844] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[858603.582975] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[859622.355657] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[859622.367826] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[860244.382730] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[860244.394849] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[861645.062139] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[861645.074287] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[862937.632134] Lustre: atlas1-MDT0000: Connection restored to 39548057-49f5-ac29-0429-a9db1f7e3466 (at 8100@gni100) <6>[862938.181572] Lustre: atlas1-MDT0000: Connection restored to d663ca82-d520-a549-5ee1-bc6f59f50f4a (at 4484@gni100) <6>[862938.193518] Lustre: Skipped 3 previous similar messages <6>[862941.436849] Lustre: atlas1-MDT0000: Connection restored to 1cfd0da0-7427-4e3e-7b46-95aa1d9dc097 (at 16914@gni100) <6>[862941.448870] Lustre: Skipped 3 previous similar messages <6>[862943.966114] Lustre: atlas1-MDT0000: Connection restored to 38c9a420-ad5a-0764-7597-b397e5adb240 (at 11856@gni100) <6>[862943.978136] Lustre: Skipped 8 previous similar messages <6>[862947.982914] Lustre: atlas1-MDT0000: Connection restored to 0a3720b7-0740-9091-6aa5-a6ec67a85e8b (at 7532@gni100) <6>[862947.994851] Lustre: Skipped 5 previous similar messages <6>[862956.334601] Lustre: atlas1-MDT0000: Connection restored to 4f305a8c-4967-b18f-9d4e-0404cbbbddb0 (at 16973@gni100) <6>[862956.346635] Lustre: Skipped 17 previous similar messages <4>[863133.415893] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[863133.428036] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[863133.440471] Lustre: Skipped 2 previous similar messages <4>[864142.975717] Lustre: atlas1-MDT0000: Client dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) reconnecting <6>[864142.988074] Lustre: atlas1-MDT0000: Connection restored to dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) <6>[864151.364538] Lustre: atlas1-MDT0000: Connection restored to 444fde80-2760-0301-0d58-59b83b91c44f (at 12411@gni100) <4>[864815.242938] Lustre: atlas1-MDT0000: Client dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) reconnecting <6>[864815.255271] Lustre: atlas1-MDT0000: Connection restored to dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) <6>[864815.267890] Lustre: Skipped 8 previous similar messages <4>[864883.044550] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[864883.056714] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[865473.382119] Lustre: atlas1-MDT0000: Client dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) reconnecting <6>[865473.394457] Lustre: atlas1-MDT0000: Connection restored to dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) <4>[865573.193588] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[865573.205731] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[866191.138051] Lustre: atlas1-MDT0000: Client dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) reconnecting <6>[866191.150423] Lustre: atlas1-MDT0000: Connection restored to dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) <4>[867181.043774] Lustre: atlas1-MDT0000: Client dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) reconnecting <6>[867181.056113] Lustre: atlas1-MDT0000: Connection restored to dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) <4>[867365.906260] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[867365.918415] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[867939.458589] Lustre: atlas1-MDT0000: Client dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) reconnecting <6>[867939.470956] Lustre: atlas1-MDT0000: Connection restored to dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) <4>[868512.289781] Lustre: atlas1-MDT0000: Client dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) reconnecting <6>[868512.302094] Lustre: atlas1-MDT0000: Connection restored to dcce02bb-8423-fc33-b18d-a53ba2b4aa52 (at 10.36.205.200@o2ib) <4>[869002.929735] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[869002.941881] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[869883.830287] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[869883.842433] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[870483.785397] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[870483.797579] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[870817.668460] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[870820.761138] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[870822.778628] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[871060.159070] Lustre: atlas1-MDT0000: Client 3e699be8-e104-6725-386e-8ca806ac5e13 (at 10.36.202.37@o2ib) reconnecting <6>[871060.171334] Lustre: atlas1-MDT0000: Connection restored to 3e699be8-e104-6725-386e-8ca806ac5e13 (at 10.36.202.37@o2ib) <4>[871108.524425] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[871108.536583] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[871758.777350] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[871758.789525] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[872495.668456] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[872495.680611] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[873203.821478] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[873203.833633] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[873845.969792] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[873845.981915] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[874509.226739] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[874509.238887] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[875189.296763] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[875189.308890] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[875937.131868] Lustre: atlas1-MDT0000: Client 3e699be8-e104-6725-386e-8ca806ac5e13 (at 10.36.202.37@o2ib) reconnecting <6>[875937.144105] Lustre: atlas1-MDT0000: Connection restored to 3e699be8-e104-6725-386e-8ca806ac5e13 (at 10.36.202.37@o2ib) <4>[876392.575937] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[876392.588099] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[877117.312351] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[877117.324485] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[877809.296112] Lustre: atlas1-MDT0000: Client 3e699be8-e104-6725-386e-8ca806ac5e13 (at 10.36.202.37@o2ib) reconnecting <6>[877809.308330] Lustre: atlas1-MDT0000: Connection restored to 3e699be8-e104-6725-386e-8ca806ac5e13 (at 10.36.202.37@o2ib) <4>[878243.084354] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[878243.096497] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[878848.231051] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[878848.243219] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[879567.686497] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[879567.698684] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[880494.351232] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[880494.363381] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[881283.991225] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[881284.003415] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[882030.824121] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[882030.836265] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[882602.865244] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[882602.877393] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[883204.738748] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[883204.750915] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[883981.104204] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[883981.116354] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[884770.687021] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[884770.699178] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[885412.218594] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[885412.230744] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[886585.517162] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[886585.529298] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[887211.374339] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[887211.386463] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[887805.623727] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[887805.635879] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[888524.670073] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[888524.682231] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[889355.514189] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[889355.526354] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[889979.870461] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[889979.882590] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[890766.389581] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[890766.401713] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[891307.284517] Lustre: atlas1-MDT0000: haven't heard from client 03a3ee4b-080e-ce9e-fb1e-cee33e790968 (at 7505@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0b86b400, cur 1487386409 expire 1487385509 last 1487385057 <4>[891307.309831] Lustre: Skipped 3 previous similar messages <3>[891307.316054] LustreError: 15624:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) ### client (nid 7505@gni100) failed to reply to blocking AST (req status 0 rc -5), evict it ns: mdt-atlas1-MDT0000_UUID lock: ffff881011b90bc0/0xd94b67485e80aac2 lrc: 5/0,0 mode: CR/CR res: [0x20039df80:0x8f6:0x0].0x0 bits 0x9 rrc: 4 type: IBT flags: 0x50200400000020 nid: 7505@gni100 remote: 0xb2eb285953366aad expref: 12 pid: 15482 timeout: 5186316038 lvb_type: 0 <3>[891307.360852] LustreError: 138-a: atlas1-MDT0000: A client on nid 7505@gni100 was evicted due to a lock blocking callback time out: rc -5 <4>[891445.081531] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[891445.093664] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[892099.750559] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[892099.762791] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[892680.512803] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[892680.524946] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[893261.800064] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[893261.812210] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[893838.770392] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[893838.782533] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[894638.628255] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[894638.640404] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[895432.970172] Lustre: atlas1-MDT0000: Connection restored to 4aa3560d-76c8-5d0c-df70-1f0b88ce9985 (at 10.36.202.21@o2ib) <4>[895526.967290] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[895526.979441] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[895526.991909] Lustre: Skipped 3 previous similar messages <4>[895857.278516] Lustre: atlas1-MDT0000: haven't heard from client 080b6494-df93-adc3-037b-a067e8469856 (at 14090@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0a684000, cur 1487390959 expire 1487390059 last 1487389607 <4>[896184.971292] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[896184.983444] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[896796.824528] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[896796.836693] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[897406.698900] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[897406.711048] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[898117.961796] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[898117.973929] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[898902.520992] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[898902.533157] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[899636.411233] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[899636.423373] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[900283.591257] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[900283.603400] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[901011.907331] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[901011.919498] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[901655.946424] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[901655.958584] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[902596.791048] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[902596.803196] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[903077.264495] Lustre: atlas1-MDT0000: haven't heard from client 55324edd-6b9f-2591-9d12-1f883086bcf8 (at 4433@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0badbc00, cur 1487398179 expire 1487397279 last 1487396827 <4>[903683.109601] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[903683.121756] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[904713.902118] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[904713.914305] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[905362.848324] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[905362.860491] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[906689.007505] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[906689.019639] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[907324.649308] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[907324.661461] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[907994.479035] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[907994.491189] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[908625.914898] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[908625.927031] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[909299.884978] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[909299.897092] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[909873.092820] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[909873.104965] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[910476.792827] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[910476.804988] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[911115.261924] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[911115.274084] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[911810.081833] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[911810.093961] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[912492.553018] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[912492.565177] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[913283.974201] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[913283.986382] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[913858.343317] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[913858.355490] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[914493.869692] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[914493.881847] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[915155.772793] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[915155.784975] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[915729.095139] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[915729.107274] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[916331.065421] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[916331.077581] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[916940.992611] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[916941.004746] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[917545.331059] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[917545.343213] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[918254.052006] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[918254.064250] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[919286.664176] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[919286.676333] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[919983.043565] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[919983.055742] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[920637.588268] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[920637.600409] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[921513.930797] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[921513.942945] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[922123.577760] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[922123.589899] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[922818.879439] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[922818.891595] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[923851.034017] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[923851.046174] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[924543.350191] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[924543.362330] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[925149.419186] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[925149.431339] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[925901.512856] Lustre: atlas1-MDT0000: Connection restored to 10.36.226.117@o2ib (at 0@lo) <6>[925901.522449] Lustre: client wants to enable acl, but mdt not! <4>[925901.708182] Lustre: Mounted atlas1-client <4>[925974.640218] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[925974.652371] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[926755.077828] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[926755.089954] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[927480.253694] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[927480.265823] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[928284.938098] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[928284.950228] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[928425.862288] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[928429.163908] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[928431.174428] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[929005.229125] Lustre: atlas1-MDT0000: haven't heard from client eba574c5-1953-c651-8e16-b137b2b2b2c0 (at 296@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f11adb000, cur 1487424107 expire 1487423207 last 1487422755 <4>[929081.499019] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[929081.511146] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[929862.990992] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[929863.003128] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[930739.171714] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[930739.183868] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[930883.830678] Lustre: atlas1-MDT0000: Connection restored to fd2da7bd-b140-c1e4-dd65-67e13423d334 (at 10.39.232.65@o2ib6) <6>[930904.330458] Lustre: atlas1-MDT0000: Connection restored to eb8a3dc7-63bb-a9f0-32e9-29e7dcf41878 (at 10.39.232.88@o2ib6) <4>[931348.461692] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[931348.473838] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[931374.108392] Lustre: Unmounted atlas1-client <4>[931936.281803] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[931936.293955] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[932572.106130] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[932572.118278] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[933182.444535] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[933182.456685] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[933811.087774] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[933811.099925] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[934494.071038] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[934494.083188] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[934627.221667] Lustre: atlas1-MDT0000: haven't heard from client fcbcd241-257b-0ff6-491e-bf0a47e146e5 (at 8749@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0af13800, cur 1487429729 expire 1487428829 last 1487428377 <4>[935078.220958] Lustre: atlas1-MDT0000: haven't heard from client 52732c69-f127-64ec-9da8-9e2df6913513 (at 8547@gni100) in 909 seconds. I think it's dead, and I am evicting it. exp ffff883f0a399400, cur 1487430180 expire 1487429280 last 1487429271 <4>[935078.246069] Lustre: Skipped 2 previous similar messages <4>[935089.719450] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[935089.731696] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[935785.573891] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[935785.586015] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[936391.439598] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[936391.451731] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[937018.644285] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[937018.656428] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[937628.086812] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[937628.099008] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[938231.258782] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[938231.270908] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[938820.003876] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[938820.016016] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[939478.808026] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[939478.820187] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[940063.114341] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[940063.126520] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[940711.886547] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[940711.898707] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[941421.494498] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[941421.506620] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[942081.988394] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[942082.000524] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[942813.043589] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[942813.055732] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[942821.130357] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[942824.140734] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[942826.205005] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[943437.366655] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[943437.378820] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[944126.052562] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[944126.064691] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[944712.503045] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[944712.515215] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[945406.890984] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[945406.903153] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[945503.233889] Lustre: atlas1-MDT0000: haven't heard from client 7c6cbb6f-5d08-9820-c38b-2cc5e9aa8e68 (at 7005@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f100e4000, cur 1487440605 expire 1487439705 last 1487439253 <4>[945988.731085] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[945988.743227] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[946914.410883] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[946914.423029] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[947585.312902] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[947585.325045] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[948195.162074] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[948195.174225] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[948323.570547] Lustre: atlas1-MDT0000: Client 2b2eb9de-8503-4228-7e84-55df1d6dcdb5 (at 15701@gni100) reconnecting <6>[948323.582314] Lustre: atlas1-MDT0000: Connection restored to 2b2eb9de-8503-4228-7e84-55df1d6dcdb5 (at 15701@gni100) <4>[949099.032730] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[949099.044863] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[949700.203910] Lustre: atlas1-MDT0000: haven't heard from client ed2de2dc-c51c-69f9-f497-bed4b683e66f (at 18643@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f1024fc00, cur 1487444802 expire 1487443902 last 1487443450 <4>[949713.023872] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[949713.036013] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[950530.092863] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[950530.105037] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[951144.884093] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[951144.896220] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[951912.112095] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[951912.124250] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[952690.562653] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[952690.574792] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[953305.583865] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[953305.596056] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[954140.387669] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[954140.399822] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[955309.685051] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[955309.697211] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[955583.215631] Lustre: atlas1-MDT0000: haven't heard from client 2ad9b73a-46cd-5955-9db7-ee79fc792d2b (at 455@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0abc5000, cur 1487450685 expire 1487449785 last 1487449333 <4>[955967.019183] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[955967.031307] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[956583.032307] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[956583.044434] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[957218.127522] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[957221.144727] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[957292.778528] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[957292.790665] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[957948.099411] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[957948.111544] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[958778.913224] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[958778.925407] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[959390.708563] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[959390.720695] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[960308.187579] Lustre: atlas1-MDT0000: haven't heard from client 835fc80e-24cd-ca04-5671-119fc1c0aec4 (at 5031@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f104a0400, cur 1487455410 expire 1487454510 last 1487454058 <4>[960308.212842] Lustre: Skipped 1 previous similar message <4>[960451.768156] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[960451.780329] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[961251.678135] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[961251.690271] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[961904.733223] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[961904.745471] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[962642.821162] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[962642.833321] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[963240.711207] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[963240.723344] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[963933.146601] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[963933.158734] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[964519.356699] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[964519.368819] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[965095.117956] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[965095.130089] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[965899.199868] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[965899.212028] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[966550.854053] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[966550.866265] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[966838.177971] Lustre: atlas1-MDT0000: haven't heard from client f672ff2f-67bc-a293-9b2a-1fd403c7b5e7 (at 18927@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f10b1e000, cur 1487461940 expire 1487461040 last 1487460588 <4>[967287.055037] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[967287.067154] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[968000.353946] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[968000.366092] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[968599.613208] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[968599.625347] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[969266.394227] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[969266.411647] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[969974.532289] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[969974.544443] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[970547.081329] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[970547.093461] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[971153.818565] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[971153.830700] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[971724.496623] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[971724.508747] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[972294.272975] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[972294.285100] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[972885.908898] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[972885.921033] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[973472.917816] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[973472.937682] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[974191.808018] Lustre: atlas1-MDT0000: Client 113d0f7c-46fc-dca1-44a1-44c633d99411 (at 12546@gni100) reconnecting <6>[974191.819760] Lustre: atlas1-MDT0000: Connection restored to 113d0f7c-46fc-dca1-44a1-44c633d99411 (at 12546@gni100) <4>[974196.281939] Lustre: atlas1-MDT0000: Client 3dc86947-5dbb-7e50-7042-66104957a24a (at 2621@gni100) reconnecting <4>[974196.293569] Lustre: Skipped 2 previous similar messages <6>[974196.299744] Lustre: atlas1-MDT0000: Connection restored to 3dc86947-5dbb-7e50-7042-66104957a24a (at 2621@gni100) <6>[974196.311662] Lustre: Skipped 2 previous similar messages <4>[974265.928725] Lustre: atlas1-MDT0000: Client 91525f2d-b2ae-ed5b-ac6f-a2c959b9cb0e (at 9229@gni100) reconnecting <6>[974265.940381] Lustre: atlas1-MDT0000: Connection restored to 91525f2d-b2ae-ed5b-ac6f-a2c959b9cb0e (at 9229@gni100) <4>[974728.034160] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[974728.046298] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[975342.307590] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[975342.319770] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[977086.851912] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[977086.864046] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[977712.026168] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[977712.038302] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[978401.753316] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[978401.765453] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[979065.361219] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[979065.373347] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[979633.986418] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[979633.998549] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[980259.427765] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[980259.439891] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[980853.346802] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[980853.358924] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[981522.066818] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[981522.078954] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[982090.090090] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[982090.102209] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[982659.155313] Lustre: atlas1-MDT0000: haven't heard from client f9894e79-855a-e7b9-1ce5-ca6a293f3050 (at 11255@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0d740400, cur 1487477761 expire 1487476861 last 1487476409 <4>[982674.415218] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[982674.427352] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[983245.794348] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[983245.806484] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[983816.594750] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[983816.606887] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[984584.125288] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[984584.137435] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[985155.402298] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[985155.414416] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[985856.303169] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[985856.315297] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[986451.050382] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[986451.062525] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[987020.307383] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[987020.319556] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[987587.933766] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[987587.945901] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[988158.460169] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[988158.472299] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[988747.976849] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[988747.988984] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[989316.096407] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[989316.108581] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[989887.211973] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[989887.224099] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[990455.878742] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[990455.890868] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[991025.067312] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[991025.079463] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[991605.501130] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[991605.513253] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[992176.747565] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[992176.759690] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[992791.731295] Lustre: atlas1-MDT0000: Client e9795983-8060-8212-7ac2-42d15a932a8d (at 6361@gni100) reconnecting <4>[992791.742922] Lustre: Skipped 4 previous similar messages <6>[992791.749134] Lustre: atlas1-MDT0000: Connection restored to e9795983-8060-8212-7ac2-42d15a932a8d (at 6361@gni100) <6>[992791.761061] Lustre: Skipped 4 previous similar messages <4>[993057.512502] Lustre: 15982:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1487487592/real 1487487592] req@ffff8828395e9680 x1558710776855572/t0(0) o104->atlas1-MDT0000@4822@gni100:15/16 lens 296/224 e 0 to 1 dl 1487488159 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[993057.543802] Lustre: 15982:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 70249165 previous similar messages <3>[993057.556077] LustreError: 15982:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) ### client (nid 6409@gni100) failed to reply to blocking AST (req status 0 rc -110), evict it ns: mdt-atlas1-MDT0000_UUID lock: ffff882bb1b02880/0xd94b674a3acf0144 lrc: 4/0,0 mode: PR/PR res: [0x2002e1c31:0x1d947:0x0].0x0 bits 0x13 rrc: 18 type: IBT flags: 0x60200400000020 nid: 6409@gni100 remote: 0xb3916f6a037a4afb expref: 22 pid: 15680 timeout: 5287532957 lvb_type: 0 <3>[993057.601468] LustreError: 15982:0:(ldlm_lockd.c:342:waiting_locks_callback()) ### lock callback timer expired after 567s: evicting client at 6409@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff882bb1b02880/0xd94b674a3acf0144 lrc: 4/0,0 mode: PR/PR res: [0x2002e1c31:0x1d947:0x0].0x0 bits 0x13 rrc: 18 type: IBT flags: 0x60200400000020 nid: 6409@gni100 remote: 0xb3916f6a037a4afb expref: 22 pid: 15680 timeout: 5287532957 lvb_type: 0 <3>[993057.650645] LustreError: 138-a: atlas1-MDT0000: A client on nid 6409@gni100 was evicted due to a lock blocking callback time out: rc -110 <4>[993085.493400] Lustre: 16359:0:(service.c:1336:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[993085.493401] req@ffff881c0efa60c0 x1558697370553820/t0(0) o101->ae491d26-39bc-3921-cc4b-f622bbfacda7@90@gni100:27/0 lens 4848/3512 e 12 to 0 dl 1487488192 ref 2 fl Interpret:/0/0 rc 0/0 <4>[993154.493305] Lustre: 16366:0:(service.c:1336:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[993154.493306] req@ffff883b56b9f680 x1558697370553828/t0(0) o101->ae491d26-39bc-3921-cc4b-f622bbfacda7@90@gni100:96/0 lens 704/3384 e 4 to 0 dl 1487488261 ref 2 fl Interpret:/0/0 rc 0/0 <3>[993309.348094] LustreError: 15978:0:(ldlm_request.c:106:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1487487661, 750s ago); not entering recovery in server code, just going back to sleep ns: mdt-atlas1-MDT0000_UUID lock: ffff8838044f8280/0xd94b674a4d2f60c0 lrc: 3/1,0 mode: --/PR res: [0x2002e1c31:0x1d947:0x0].0x0 bits 0x13 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 15978 timeout: 0 lvb_type: 0 <4>[993414.852938] Lustre: 15789:0:(service.c:1336:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[993414.852939] req@ffff882834e079c0 x1549208003017568/t0(0) o101->ab4e429d-1b8c-bbdb-2826-67af15b260c0@10.36.205.206@o2ib:356/0 lens 704/3384 e 12 to 0 dl 1487488521 ref 2 fl Interpret:/0/0 rc 0/0 <3>[993569.849729] LustreError: 15910:0:(ldlm_request.c:106:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1487487921, 750s ago); not entering recovery in server code, just going back to sleep ns: mdt-atlas1-MDT0000_UUID lock: ffff880eea483a80/0xd94b674a4f1b01fd lrc: 3/1,0 mode: --/PR res: [0x2002e1c31:0x1d947:0x0].0x0 bits 0x13 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 15910 timeout: 0 lvb_type: 0 <4>[993624.666732] Lustre: 15982:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1487488159/real 1487488159] req@ffff882a715d26c0 x1558710776856116/t0(0) o104->atlas1-MDT0000@6358@gni100:15/16 lens 296/224 e 0 to 1 dl 1487488726 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 <4>[993624.698059] Lustre: 15982:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 11 previous similar messages <3>[993624.709668] LustreError: 15982:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) ### client (nid 6358@gni100) failed to reply to blocking AST (req status 0 rc -110), evict it ns: mdt-atlas1-MDT0000_UUID lock: ffff88205c207100/0xd94b674a3acf0cd5 lrc: 4/0,0 mode: PR/PR res: [0x2002e1c31:0x1d947:0x0].0x0 bits 0x13 rrc: 7 type: IBT flags: 0x60200400000020 nid: 6358@gni100 remote: 0xbfca7c99dceb782f expref: 22 pid: 16131 timeout: 5288625018 lvb_type: 0 <3>[993624.754983] LustreError: 15982:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) Skipped 1 previous similar message <3>[993624.766740] LustreError: 138-a: atlas1-MDT0000: A client on nid 6358@gni100 was evicted due to a lock blocking callback time out: rc -110 <3>[993624.781105] LustreError: Skipped 1 previous similar message <4>[993624.789100] Lustre: 15982:0:(service.c:2097:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:534s); client may timeout. req@ffff881c0efa60c0 x1558697370553820/t648397696747(0) o101->ae491d26-39bc-3921-cc4b-f622bbfacda7@90@gni100:27/0 lens 4848/672 e 12 to 0 dl 1487488192 ref 1 fl Complete:/0/0 rc 0/0 <4>[993624.822774] Lustre: 15982:0:(service.c:2097:ptlrpc_server_handle_request()) Skipped 2 previous similar messages <4>[994215.085489] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[994215.097604] Lustre: Skipped 18 previous similar messages <6>[994215.103885] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[994215.116363] Lustre: Skipped 20 previous similar messages <4>[994784.242373] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[994784.254494] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[995352.959893] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[995352.972036] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[995922.223839] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[995922.235968] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[997063.088174] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[997063.100308] Lustre: Skipped 1 previous similar message <6>[997063.106410] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[997063.118843] Lustre: Skipped 1 previous similar message <4>[998202.715692] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[998202.727806] Lustre: Skipped 1 previous similar message <6>[998202.733903] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[998202.746333] Lustre: Skipped 1 previous similar message <4>[999356.476078] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[999356.488285] Lustre: Skipped 1 previous similar message <6>[999356.494375] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[999356.506848] Lustre: Skipped 1 previous similar message <4>[999967.925062] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[999967.937200] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1001104.245463] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[1001104.257675] Lustre: Skipped 1 previous similar message <6>[1001104.263875] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[1001104.276430] Lustre: Skipped 1 previous similar message <4>[1001875.670432] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1001875.682660] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1003145.916634] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[1003145.928859] Lustre: Skipped 1 previous similar message <6>[1003145.935054] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[1003145.947573] Lustre: Skipped 1 previous similar message <4>[1003737.552915] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1003737.565169] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1004307.318987] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1004307.331226] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1004878.938279] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1004878.950516] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1005878.324145] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1005878.336407] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1006632.849106] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1006632.861330] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1007855.901914] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <4>[1007855.914143] Lustre: Skipped 1 previous similar message <6>[1007855.920330] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <6>[1007855.932873] Lustre: Skipped 1 previous similar message <4>[1008446.621864] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1008446.634106] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1009796.005257] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1009796.017503] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <3>[1010476.090716] LustreError: 15945:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20039fe88:0xd2ac:0x0]: rc = -2 <4>[1011182.271418] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1011182.283654] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1012815.112917] Lustre: atlas1-MDT0000: haven't heard from client 3b409862-ed12-1f21-587f-195f6d282c72 (at 17120@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f101bdc00, cur 1487507917 expire 1487507017 last 1487506565 <4>[1012928.431265] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1012928.443495] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1014654.832793] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1014654.845041] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1014822.516726] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1014825.501842] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1014827.818422] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1022663.098703] Lustre: atlas1-MDT0000: haven't heard from client 3360bddb-2f0d-ca7d-753f-8e58e59b59dc (at 13129@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0e2d7800, cur 1487517765 expire 1487516865 last 1487516413 <3>[1022663.124113] LustreError: 15796:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) ### client (nid 13129@gni100) failed to reply to blocking AST (req status 0 rc -5), evict it ns: mdt-atlas1-MDT0000_UUID lock: ffff883ca38e19c0/0xd94b674ba415e9f9 lrc: 4/0,0 mode: CR/CR res: [0x20039b42a:0x8ea:0x0].0x0 bits 0x9 rrc: 3 type: IBT flags: 0x60200400000020 nid: 13129@gni100 remote: 0xc972975160adc2b3 expref: 12 pid: 15650 timeout: 5318133925 lvb_type: 0 <3>[1022663.169158] LustreError: 15796:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) Skipped 2 previous similar messages <3>[1022663.186246] LustreError: 138-a: atlas1-MDT0000: A client on nid 13129@gni100 was evicted due to a lock blocking callback time out: rc -5 <3>[1022663.200590] LustreError: Skipped 2 previous similar messages <4>[1023114.098099] Lustre: atlas1-MDT0000: haven't heard from client 3b95e27c-72bb-7295-054f-751d220edd08 (at 5717@gni100) in 906 seconds. I think it's dead, and I am evicting it. exp ffff883f0b9eac00, cur 1487518216 expire 1487517316 last 1487517310 <4>[1027042.092372] Lustre: atlas1-MDT0000: haven't heard from client 2988552a-e96d-b2f0-1984-4f767b8b3ee3 (at 2671@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0cc89c00, cur 1487522144 expire 1487521244 last 1487520792 <4>[1029222.089204] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1029225.106309] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1029227.250144] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1029946.088336] Lustre: atlas1-MDT0000: haven't heard from client 17b0c88f-3622-983e-5fba-2f3b938f23ef (at 17652@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883edb8a8800, cur 1487525048 expire 1487524148 last 1487523696 <6>[1030301.382266] Lustre: atlas1-MDT0000: Connection restored to 10.36.226.117@o2ib (at 0@lo) <6>[1030301.391944] Lustre: client wants to enable acl, but mdt not! <4>[1030301.580041] Lustre: Mounted atlas1-client <4>[1036129.792285] Lustre: Unmounted atlas1-client <4>[1036271.983137] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1036271.995396] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1036857.619542] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1036857.631845] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1037533.828655] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1037533.840899] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1037627.077252] Lustre: atlas1-MDT0000: haven't heard from client 3723cfd0-72ce-a4d8-032c-63fad6cfe721 (at 18907@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0ccb9c00, cur 1487532729 expire 1487531829 last 1487531377 <4>[1038107.807683] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1038107.819908] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1039812.079182] Lustre: atlas1-MDT0000: haven't heard from client d6df176c-9917-5519-4344-52e09dd6f522 (at 2903@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0c6e9800, cur 1487534914 expire 1487534014 last 1487533562 <4>[1043623.086352] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1043626.104927] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1043628.442353] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <3>[1048252.955976] LustreError: 15659:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20039ef25:0x7232:0x0]: rc = -2 <4>[1049582.067668] Lustre: atlas1-MDT0000: haven't heard from client 9b6b1578-8d30-b45a-a732-24a03740ffae (at 2956@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f10f80400, cur 1487544684 expire 1487543784 last 1487543332 <4>[1052434.068154] Lustre: atlas1-MDT0000: haven't heard from client c4437712-7fa1-86f3-8aad-3274f4afa9a2 (at 15003@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0baa6800, cur 1487547536 expire 1487546636 last 1487546184 <4>[1057480.553603] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1057480.565838] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1058091.773658] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1058091.785885] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1059331.735853] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1059331.748101] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1060086.188862] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1060086.201096] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1062490.767154] Lustre: atlas1-MDT0000: Client 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) reconnecting <6>[1062490.779419] Lustre: atlas1-MDT0000: Connection restored to 49207f95-72eb-913e-ab93-3a579be04014 (at 10.36.225.5@o2ib) <4>[1069460.034680] Lustre: atlas1-MDT0000: haven't heard from client 86863d15-f820-b36e-434e-046c11633fc9 (at 8613@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f1054e000, cur 1487564562 expire 1487563662 last 1487563210 <6>[1073818.481554] Lustre: atlas1-MDT0000: Connection restored to c967d511-d644-3c9e-356a-7d98c2fc2ad9 (at 10.38.144.244@o2ib5) <3>[1074742.781301] LustreError: 15672:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) ### client (nid 10.38.144.244@o2ib5) returned error from blocking AST (req status -107 rc -107), evict it ns: mdt-atlas1-MDT0000_UUID lock: ffff8827046cb580/0xd94b674d7a102d33 lrc: 4/0,0 mode: PR/PR res: [0x20024229d:0x10cba:0x0].0x0 bits 0x13 rrc: 26 type: IBT flags: 0x60200400000020 nid: 10.38.144.244@o2ib5 remote: 0x7455b5b507e905c0 expref: 128 pid: 15595 timeout: 5369785357 lvb_type: 0 <3>[1074742.828785] LustreError: 138-a: atlas1-MDT0000: A client on nid 10.38.144.244@o2ib5 was evicted due to a lock blocking callback time out: rc -107 <4>[1084379.016104] Lustre: atlas1-MDT0000: haven't heard from client 5f76aa71-6ce7-99cc-2bc8-614378c65459 (at 245@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0bdfe800, cur 1487579481 expire 1487578581 last 1487578129 <4>[1094936.999317] Lustre: atlas1-MDT0000: haven't heard from client 44ba422f-8f15-e8e0-87af-586bb036a880 (at 5189@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f11224c00, cur 1487590039 expire 1487589139 last 1487588687 <3>[1095975.075638] LustreError: 15816:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a0e5d:0x14701:0x0]: rc = -2 <4>[1100411.991015] Lustre: atlas1-MDT0000: haven't heard from client a21aa519-5782-086e-b8d7-1b6be4162ec5 (at 16296@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f111da800, cur 1487595514 expire 1487594614 last 1487594162 <4>[1101226.242242] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1101230.154975] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1101232.212296] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1106347.982766] Lustre: atlas1-MDT0000: haven't heard from client 6f6eed79-294d-2d20-5406-f59f38797320 (at 18000@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f1284bc00, cur 1487601450 expire 1487600550 last 1487600098 <4>[1109393.978102] Lustre: atlas1-MDT0000: haven't heard from client d9bee6ab-7dc1-8cad-6c03-b4c38f996e1a (at 2019@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f100e4c00, cur 1487604496 expire 1487603596 last 1487603144 <4>[1115620.149433] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1115623.095609] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1115625.159170] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1118106.968883] Lustre: atlas1-MDT0000: haven't heard from client 2f4b2f89-d52f-b29a-dc9e-cc322edc5f21 (at 970@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0de4d400, cur 1487613209 expire 1487612309 last 1487611857 <3>[1120726.885609] LustreError: 15506:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a0553:0x3d:0x0]: rc = -2 <4>[1126712.954116] Lustre: atlas1-MDT0000: haven't heard from client 512da01e-d57c-8b68-aeeb-c0808ef4fe87 (at 17454@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0eab2c00, cur 1487621815 expire 1487620915 last 1487620463 <4>[1130019.082671] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1130022.096486] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1130024.147093] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1155091.913996] Lustre: atlas1-MDT0000: haven't heard from client 243a449a-0c5d-2f02-ac42-43a39ee33e36 (at 10251@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0ffd3c00, cur 1487650194 expire 1487649294 last 1487648842 <4>[1156829.911515] Lustre: atlas1-MDT0000: haven't heard from client 254b1877-659d-b226-8fe4-553f11ca69d1 (at 16927@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0fa33c00, cur 1487651932 expire 1487651032 last 1487650580 <4>[1162034.337525] Lustre: atlas1-MDT0000: Client 0c782632-bf75-f869-2387-5c4153b90830 (at 10.38.144.46@o2ib5) reconnecting <6>[1162034.349955] Lustre: atlas1-MDT0000: Connection restored to 3d5f94fe-1f5d-e88e-3509-de3ccd43d465 (at 10.38.144.46@o2ib5) <4>[1181524.119911] Lustre: 16243:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1487676059/real 1487676059] req@ffff883c5c7dd380 x1558713171029164/t0(0) o104->atlas1-MDT0000@12409@gni100:15/16 lens 296/224 e 0 to 1 dl 1487676626 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[1181524.151405] Lustre: 16243:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 2 previous similar messages <4>[1181551.909811] Lustre: 16426:0:(service.c:1336:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[1181551.909812] req@ffff88330f6c80c0 x1558697426778512/t0(0) o36->835099db-203b-44fa-2e02-0c72784e505c@9298@gni100:499/0 lens 616/3128 e 12 to 0 dl 1487676659 ref 2 fl Interpret:/0/0 rc 0/0 <4>[1181806.952484] Lustre: atlas1-MDT0000: Client 835099db-203b-44fa-2e02-0c72784e505c (at 9298@gni100) reconnecting <6>[1181806.964240] Lustre: atlas1-MDT0000: Connection restored to 835099db-203b-44fa-2e02-0c72784e505c (at 9298@gni100) <4>[1182091.194149] Lustre: 16243:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1487676626/real 1487676626] req@ffff883c5c7dd380 x1558713171029164/t0(0) o104->atlas1-MDT0000@12409@gni100:15/16 lens 296/224 e 0 to 1 dl 1487677193 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 <3>[1182091.225746] LustreError: 16243:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) ### client (nid 12409@gni100) failed to reply to blocking AST (req status 0 rc -110), evict it ns: mdt-atlas1-MDT0000_UUID lock: ffff880ffdebc4c0/0xd94b6754570b3bb4 lrc: 4/0,0 mode: PR/PR res: [0x2003a1c20:0xf0b5:0x0].0x0 bits 0x13 rrc: 3 type: IBT flags: 0x60200400000020 nid: 12409@gni100 remote: 0x621378064f6a491c expref: 14 pid: 15493 timeout: 5477091889 lvb_type: 0 <3>[1182091.271207] LustreError: 138-a: atlas1-MDT0000: A client on nid 12409@gni100 was evicted due to a lock blocking callback time out: rc -110 <4>[1182091.286800] Lustre: 16243:0:(service.c:2097:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:534s); client may timeout. req@ffff88330f6c80c0 x1558697426778512/t649995942991(0) o36->835099db-203b-44fa-2e02-0c72784e505c@9298@gni100:499/0 lens 616/424 e 12 to 0 dl 1487676659 ref 1 fl Complete:/0/0 rc 0/0 <6>[1185101.284679] Lustre: atlas1-MDT0000: Connection restored to a8db0739-e1be-1713-0686-751b1bca8794 (at 0@lo) <6>[1185101.296121] Lustre: client wants to enable acl, but mdt not! <4>[1185101.483866] Lustre: Mounted atlas1-client <4>[1187620.545908] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1187623.589967] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1187625.602033] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1188163.867642] Lustre: atlas1-MDT0000: haven't heard from client 05c8fc2d-ee91-ac6b-0ea2-d9f871b9154b (at 14526@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f16cdd400, cur 1487683266 expire 1487682366 last 1487681914 <4>[1190934.966636] Lustre: Unmounted atlas1-client <4>[1191047.863647] Lustre: atlas1-MDT0000: haven't heard from client 8f5bf2db-6559-4285-44c2-7ec6aee9c704 (at 9398@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0d715c00, cur 1487686150 expire 1487685250 last 1487684798 <6>[1193042.248144] Lustre: atlas1-MDT0000: Connection restored to 642389ce-7e7b-4fbe-a401-3a7fc148643b (at 10.39.232.65@o2ib6) <4>[1202018.771997] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1202021.846371] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1202023.966997] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1206548.841204] Lustre: atlas1-MDT0000: haven't heard from client 7eb21647-0f04-ece0-b4d2-fa60235a6a4f (at 9436@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f12ad9400, cur 1487701651 expire 1487700751 last 1487700299 <4>[1215674.827701] Lustre: atlas1-MDT0000: haven't heard from client fd40c450-49f3-2417-8205-4100b71306ad (at 18669@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0fd73400, cur 1487710777 expire 1487709877 last 1487709425 <4>[1216423.238596] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1216426.232780] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1216428.578680] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1223382.818984] Lustre: atlas1-MDT0000: haven't heard from client f3c11f0c-75a8-54ae-6600-fc73282cf415 (at 9261@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0f2be800, cur 1487718485 expire 1487717585 last 1487717133 <4>[1224676.818800] Lustre: atlas1-MDT0000: haven't heard from client a18ade67-e607-8d7f-c14d-2c0d2e77b192 (at 4487@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f1021cc00, cur 1487719779 expire 1487718879 last 1487718427 <4>[1237115.808349] Lustre: atlas1-MDT0000: haven't heard from client 74edea9e-190f-d45e-6b45-ec7b51a48ac5 (at 5019@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0f511000, cur 1487732218 expire 1487731318 last 1487730866 <4>[1241322.795242] Lustre: atlas1-MDT0000: haven't heard from client 0e87e257-888d-d02b-8321-bfa8bd7840b8 (at 15427@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0ab91c00, cur 1487736425 expire 1487735525 last 1487735073 <4>[1248385.791111] Lustre: atlas1-MDT0000: haven't heard from client d0330f77-c3a4-ea4e-6810-99e4f20cec0b (at 17022@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f11ae3000, cur 1487743488 expire 1487742588 last 1487742136 <4>[1258604.775875] Lustre: atlas1-MDT0000: haven't heard from client 02cfca07-abda-e6f3-c8b9-f54c9cca3cbb (at 7@gni2) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883ed6777000, cur 1487753707 expire 1487752807 last 1487752355 <4>[1265865.762936] Lustre: atlas1-MDT0000: haven't heard from client 4925b678-5adf-016a-bc33-ddacdf098ffe (at 2054@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f12efd400, cur 1487760968 expire 1487760068 last 1487759616 <4>[1274020.625420] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1274023.669810] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1274025.768297] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1275858.747907] Lustre: atlas1-MDT0000: haven't heard from client 9d574720-f141-68bc-5eb7-6814beb82c47 (at 12455@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0c5ee800, cur 1487770961 expire 1487770061 last 1487769609 <6>[1278663.224571] Lustre: atlas1-MDT0000: Connection restored to 39e96fa3-18e6-c284-0f19-c6fc95b67e87 (at 95@gni2) <4>[1279756.747862] Lustre: atlas1-MDT0000: haven't heard from client 12379f51-0928-df48-de07-8f3d916aaf8c (at 95@gni2) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883ed1bfc000, cur 1487774859 expire 1487773959 last 1487773507 <4>[1279756.772831] Lustre: Skipped 244 previous similar messages <6>[1279878.822732] Lustre: atlas1-MDT0000: Connection restored to a9e932b3-54f1-5b55-20fc-2c315ce3f9f3 (at 95@gni2) <6>[1280691.415378] Lustre: atlas1-MDT0000: Connection restored to 39e96fa3-18e6-c284-0f19-c6fc95b67e87 (at 95@gni2) <4>[1280915.739406] Lustre: atlas1-MDT0000: haven't heard from client 39e96fa3-18e6-c284-0f19-c6fc95b67e87 (at 95@gni2) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f10075c00, cur 1487776018 expire 1487775118 last 1487774666 <6>[1281499.239949] Lustre: atlas1-MDT0000: Connection restored to 64013e63-1408-70b2-9842-1915bb52fbb3 (at 95@gni2) <4>[1281831.738318] Lustre: atlas1-MDT0000: haven't heard from client a9e932b3-54f1-5b55-20fc-2c315ce3f9f3 (at 95@gni2) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883bb66e8400, cur 1487776934 expire 1487776034 last 1487775582 <6>[1282203.928435] Lustre: atlas1-MDT0000: Connection restored to 39e96fa3-18e6-c284-0f19-c6fc95b67e87 (at 95@gni2) <4>[1282282.738924] Lustre: atlas1-MDT0000: haven't heard from client 9a116078-3c3b-1e4c-25c3-e5646b4139a2 (at 95@gni2) in 1140 seconds. I think it's dead, and I am evicting it. exp ffff883ed89d9400, cur 1487777385 expire 1487776485 last 1487776245 <4>[1283301.736258] Lustre: atlas1-MDT0000: haven't heard from client 64013e63-1408-70b2-9842-1915bb52fbb3 (at 95@gni2) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883dcb463800, cur 1487778404 expire 1487777504 last 1487777052 <4>[1284613.735335] Lustre: atlas1-MDT0000: haven't heard from client abc28ce9-5657-f99e-d974-b9adf75badd6 (at 12182@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0fb32000, cur 1487779716 expire 1487778816 last 1487778364 <6>[1284727.992193] Lustre: atlas1-MDT0000: Connection restored to 794a61f1-286c-6dd8-47af-f635f0a47e69 (at 10.36.247.134@o2ib) <6>[1284740.008173] Lustre: atlas1-MDT0000: Connection restored to d8ee2fa0-e872-367c-eea8-3f3678f07918 (at 10.36.247.131@o2ib) <6>[1284740.020874] Lustre: Skipped 29 previous similar messages <6>[1287829.803667] Lustre: atlas1-MDT0000: Connection restored to 1105f478-9598-6d17-59eb-17b15fbcf185 (at 10.39.232.84@o2ib6) <6>[1287831.518530] Lustre: atlas1-MDT0000: Connection restored to cc2510c5-69a7-b5b9-9e54-e086d34c74bd (at 10.39.232.62@o2ib6) <6>[1287835.405128] Lustre: atlas1-MDT0000: Connection restored to 77c94c78-61fe-2da5-89d1-6ec4b608bdbd (at 10.39.232.76@o2ib6) <6>[1287838.571145] Lustre: atlas1-MDT0000: Connection restored to 1a22ef46-832b-ad1d-eea2-8cf385643af1 (at 10.39.232.103@o2ib6) <6>[1287838.583945] Lustre: Skipped 3 previous similar messages <6>[1287843.673804] Lustre: atlas1-MDT0000: Connection restored to ccfd6b04-19f1-d4bc-05ce-0cef5b7f45ca (at 10.39.232.75@o2ib6) <6>[1287843.686504] Lustre: Skipped 5 previous similar messages <6>[1287862.018918] Lustre: atlas1-MDT0000: Connection restored to 4d2490c2-ace2-6d3d-990c-04091fd8ba2f (at 10.39.232.89@o2ib6) <6>[1287862.031612] Lustre: Skipped 12 previous similar messages <6>[1288158.406305] Lustre: atlas1-MDT0000: Connection restored to e9b28ea4-e7cc-67e0-2115-5a0c5d5f1b23 (at 10.39.232.54@o2ib6) <4>[1288418.595500] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1288421.634218] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1288423.681731] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <3>[1290100.123205] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 16921@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff881e6c630dc0/0xd94b67579a0803bf lrc: 4/0,0 mode: PR/PR res: [0x200388719:0x11c7:0x0].0x0 bits 0x13 rrc: 81 type: IBT flags: 0x60200400000020 nid: 16921@gni100 remote: 0x3b34b6db55a5d74c expref: 31 pid: 15993 timeout: 5584767252 lvb_type: 0 <3>[1290100.166691] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) Skipped 1 previous similar message <4>[1290102.726292] Lustre: atlas1-MDT0000: haven't heard from client 900a026b-1a63-85b0-4a47-c0380a62cfc8 (at 17941@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0fb0b000, cur 1487785205 expire 1487784305 last 1487783853 <3>[1290955.882667] LustreError: 15939:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a283b:0xa44f:0x0]: rc = -2 <4>[1298530.714209] Lustre: atlas1-MDT0000: haven't heard from client 48df30bb-f7a5-a56f-f865-04601bef39e9 (at 6109@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0ccf9c00, cur 1487793633 expire 1487792733 last 1487792281 <4>[1302816.730069] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1302819.818450] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <6>[1303900.700698] Lustre: atlas1-MDT0000: Connection restored to 10.36.226.117@o2ib (at 0@lo) <6>[1303900.710359] Lustre: client wants to enable acl, but mdt not! <4>[1303900.891939] Lustre: Mounted atlas1-client <6>[1304806.360102] Lustre: atlas1-MDT0000: Connection restored to 6ccca0bf-b0d5-11dc-ace5-f2caf992032b (at 7556@gni100) <6>[1304806.372125] Lustre: Skipped 1 previous similar message <6>[1304806.893337] Lustre: atlas1-MDT0000: Connection restored to 94cdb09f-ede1-6076-50a7-c02c6da14b0f (at 12166@gni100) <6>[1304806.905458] Lustre: Skipped 11 previous similar messages <6>[1304808.027518] Lustre: atlas1-MDT0000: Connection restored to 4518ed6a-1d89-c139-7f4a-63415c0ea62b (at 9336@gni100) <6>[1304808.039669] Lustre: Skipped 24 previous similar messages <6>[1304810.539968] Lustre: atlas1-MDT0000: Connection restored to d5d159bb-a2a4-08b2-2afa-65be157f5533 (at 7806@gni100) <6>[1304810.551991] Lustre: Skipped 12 previous similar messages <6>[1304814.547856] Lustre: atlas1-MDT0000: Connection restored to fc602a9e-e12e-f11d-3316-2bb9c90af1fd (at 10869@gni100) <6>[1304814.560118] Lustre: Skipped 49 previous similar messages <6>[1304822.729958] Lustre: atlas1-MDT0000: Connection restored to 3163ce1f-7999-73d2-270d-eaceb6befebd (at 7515@gni100) <6>[1304822.741979] Lustre: Skipped 95 previous similar messages <6>[1305618.067602] Lustre: atlas1-MDT0000: Connection restored to 806beb47-dd8c-4f74-fed8-38bd1b13bd2e (at 12414@gni100) <6>[1305618.079769] Lustre: Skipped 48 previous similar messages <6>[1305625.425871] Lustre: atlas1-MDT0000: Connection restored to 432e575c-bfe9-1dda-9b62-18c087a61fa6 (at 12175@gni100) <6>[1305625.437985] Lustre: Skipped 2 previous similar messages <6>[1305630.550424] Lustre: atlas1-MDT0000: Connection restored to b9cc2b41-3754-f127-e538-b35aa925fd80 (at 12117@gni100) <6>[1305630.562541] Lustre: Skipped 8 previous similar messages <6>[1305638.571393] Lustre: atlas1-MDT0000: Connection restored to 4419e6cf-d1de-1683-3a4a-020bfa50d7cd (at 7561@gni100) <6>[1305638.588548] Lustre: Skipped 31 previous similar messages <6>[1305655.893657] Lustre: atlas1-MDT0000: Connection restored to f6ab56bf-71d6-6376-02a2-34e3b7b2cee4 (at 7793@gni100) <6>[1305655.905679] Lustre: Skipped 54 previous similar messages <6>[1305689.418882] Lustre: atlas1-MDT0000: Connection restored to f6c6b05e-7c6c-e88a-bdd7-ca8e5a789c09 (at 9332@gni100) <6>[1305689.430902] Lustre: Skipped 93 previous similar messages <4>[1307595.702054] Lustre: atlas1-MDT0000: haven't heard from client 8aca6c63-838d-8877-3ec6-9b7db6ab405a (at 17584@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f121cac00, cur 1487802698 expire 1487801798 last 1487801346 <4>[1309735.452104] Lustre: Unmounted atlas1-client <3>[1314665.050281] LustreError: 15468:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a1152:0x15e02:0x0]: rc = -2 <4>[1319538.684809] Lustre: atlas1-MDT0000: haven't heard from client 0c44e656-5790-29cf-606a-9b88f07b69fb (at 3937@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0a9ffc00, cur 1487814641 expire 1487813741 last 1487813289 <4>[1322263.680324] Lustre: atlas1-MDT0000: haven't heard from client fbe42506-b962-3290-a81a-534bddc4b7ab (at 236@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0f94ac00, cur 1487817366 expire 1487816466 last 1487816014 <3>[1326975.070779] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 6078@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff881a783e1d80/0xd94b675918b31631 lrc: 4/0,0 mode: PR/PR res: [0x200388719:0x11c7:0x0].0x0 bits 0x13 rrc: 99 type: IBT flags: 0x60200400000020 nid: 6078@gni100 remote: 0x5a78e37b0c87ec0f expref: 29 pid: 15841 timeout: 5621642136 lvb_type: 0 <3>[1355210.368991] LustreError: 15537:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a1189:0x6d0:0x0]: rc = -2 <3>[1355210.383932] LustreError: 15537:0:(mdt_handler.c:893:mdt_getattr_internal()) Skipped 1 previous similar message <4>[1360422.048737] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1360424.991279] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1361914.633941] Lustre: atlas1-MDT0000: haven't heard from client ebfeeea7-e2e7-f85f-e38a-5b84e652e375 (at 1957@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0ccaa400, cur 1487857017 expire 1487856117 last 1487855665 <6>[1362472.579084] Lustre: atlas1-MDT0000: Connection restored to 721721aa-4862-d86f-86f5-2db4a578cffb (at 10.39.232.104@o2ib6) <6>[1362472.591885] Lustre: Skipped 10 previous similar messages <6>[1362571.677325] Lustre: atlas1-MDT0000: Connection restored to 5916b1cd-c4f8-4025-2be8-697f5d6cf6e0 (at 10.39.232.57@o2ib6) <6>[1362590.279527] Lustre: atlas1-MDT0000: Connection restored to ce742519-4723-712a-be78-60e3cd9f9176 (at 10.39.232.64@o2ib6) <6>[1362590.292261] Lustre: Skipped 1 previous similar message <6>[1362640.929014] Lustre: atlas1-MDT0000: Connection restored to 4d2490c2-ace2-6d3d-990c-04091fd8ba2f (at 10.39.232.89@o2ib6) <6>[1362640.941725] Lustre: Skipped 3 previous similar messages <4>[1362874.624033] Lustre: atlas1-MDT0000: haven't heard from client 81591d8a-7626-02a0-510a-6e632304281e (at 16059@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0f048c00, cur 1487857977 expire 1487857077 last 1487856625 <6>[1363092.199100] Lustre: atlas1-MDT0000: Connection restored to 28c9f276-54db-9571-2d2b-a259b0d9fed4 (at 10.39.232.67@o2ib6) <6>[1363092.211792] Lustre: Skipped 4 previous similar messages <4>[1363325.623541] Lustre: atlas1-MDT0000: haven't heard from client 6de1ccb9-0074-a821-fd7d-5f03cf8436ed (at 10.39.232.59@o2ib6) in 1145 seconds. I think it's dead, and I am evicting it. exp ffff881454cbd800, cur 1487858428 expire 1487857528 last 1487857283 <4>[1363325.649583] Lustre: Skipped 304 previous similar messages <6>[1363328.416828] Lustre: atlas1-MDT0000: Connection restored to 9a212798-f1f4-8060-3aa8-dac397577d0c (at 10.39.232.63@o2ib6) <6>[1363683.401791] Lustre: atlas1-MDT0000: Connection restored to 28c9f276-54db-9571-2d2b-a259b0d9fed4 (at 10.39.232.67@o2ib6) <4>[1363776.627317] Lustre: atlas1-MDT0000: haven't heard from client 28c9f276-54db-9571-2d2b-a259b0d9fed4 (at 10.39.232.67@o2ib6) in 1018 seconds. I think it's dead, and I am evicting it. exp ffff880f9a91a000, cur 1487858879 expire 1487857979 last 1487857861 <4>[1363776.653306] Lustre: Skipped 10 previous similar messages <4>[1364227.622194] Lustre: atlas1-MDT0000: haven't heard from client 48a2ac57-7900-9dc5-fe13-0bea86d4b3f1 (at 10.39.232.63@o2ib6) in 1318 seconds. I think it's dead, and I am evicting it. exp ffff881454cbd000, cur 1487859330 expire 1487858430 last 1487858012 <6>[1364384.796815] Lustre: atlas1-MDT0000: Connection restored to 6a531cbc-e791-08e4-4ee1-e53ebfca191a (at 10.39.232.91@o2ib6) <6>[1364384.809514] Lustre: Skipped 1 previous similar message <4>[1364678.625089] Lustre: atlas1-MDT0000: haven't heard from client 4251577b-a95c-d941-6851-4a36e7478891 (at 10.39.232.85@o2ib6) in 1102 seconds. I think it's dead, and I am evicting it. exp ffff883ed56ad800, cur 1487859781 expire 1487858881 last 1487858679 <4>[1364678.651133] Lustre: Skipped 1 previous similar message <4>[1365129.620852] Lustre: atlas1-MDT0000: haven't heard from client 9f482375-7f52-d9dd-2b1c-740b04eb16b6 (at 10.39.232.65@o2ib6) in 1087 seconds. I think it's dead, and I am evicting it. exp ffff880db415c000, cur 1487860232 expire 1487859332 last 1487859145 <4>[1370675.613599] Lustre: atlas1-MDT0000: haven't heard from client e9b49c3f-a077-cd78-3972-8f7bbd0af58c (at 10145@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f101f3000, cur 1487865778 expire 1487864878 last 1487864426 <4>[1370675.638971] Lustre: Skipped 2 previous similar messages <4>[1372338.612501] Lustre: atlas1-MDT0000: haven't heard from client 582d729c-5de7-ece1-ae0c-8f89978f2c75 (at 15504@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0c77b400, cur 1487867441 expire 1487866541 last 1487866089 <4>[1374817.883426] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1374820.942821] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1376761.888128] Lustre: atlas1-MDT0000: Client b596c3ab-1143-379f-a187-85976899e39f (at 17818@gni100) reconnecting <6>[1376761.899980] Lustre: atlas1-MDT0000: Connection restored to b596c3ab-1143-379f-a187-85976899e39f (at 17818@gni100) <6>[1376761.912122] Lustre: Skipped 2 previous similar messages <4>[1377674.603835] Lustre: atlas1-MDT0000: haven't heard from client bff23e3f-c63b-bf98-6fe7-10614d5670ab (at 17819@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0d55c800, cur 1487872777 expire 1487871877 last 1487871425 <4>[1378125.602741] Lustre: atlas1-MDT0000: haven't heard from client 3be63a05-94d2-54f3-c5e8-8c8285d9b94a (at 18872@gni100) in 1302 seconds. I think it's dead, and I am evicting it. exp ffff883f0c86f800, cur 1487873228 expire 1487872328 last 1487871926 <4>[1379787.600433] Lustre: atlas1-MDT0000: haven't heard from client c30d951a-4f43-6c32-0e79-d7043128ea9b (at 13261@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8827d3d04400, cur 1487874890 expire 1487873990 last 1487873538 <6>[1381683.978598] Lustre: atlas1-MDT0000: Connection restored to 897ae187-dd52-5961-6ac2-1dff5b1055fc (at 5304@gni100) <6>[1381693.356389] Lustre: atlas1-MDT0000: Connection restored to b45cfebc-943c-bfa8-5677-de34d281e1c1 (at 6835@gni100) <6>[1381693.368419] Lustre: Skipped 120 previous similar messages <6>[1381712.702464] Lustre: atlas1-MDT0000: Connection restored to 45bd0b45-6942-f1b6-924a-527b1a8ce29c (at 3821@gni100) <6>[1381712.714484] Lustre: Skipped 185 previous similar messages <6>[1382718.132594] Lustre: atlas1-MDT0000: Connection restored to bf4b53fa-0893-a9aa-996f-5ec682f1de32 (at 5303@gni100) <6>[1382723.035668] Lustre: atlas1-MDT0000: Connection restored to f2a00787-bf90-50ed-b0f6-ec359d621c58 (at 5306@gni100) <6>[1382723.047702] Lustre: Skipped 22 previous similar messages <6>[1382751.986796] Lustre: atlas1-MDT0000: Connection restored to 1d5394ce-2a4f-5482-b6f6-1c93b171189c (at 6838@gni100) <6>[1382751.998823] Lustre: Skipped 71 previous similar messages <4>[1383283.595422] Lustre: atlas1-MDT0000: haven't heard from client 1e0280b9-3864-67c0-0b81-e828683e70fd (at 6047@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0f44ac00, cur 1487878386 expire 1487877486 last 1487877034 <4>[1384902.593047] Lustre: atlas1-MDT0000: haven't heard from client 2a0d3a86-93c4-558f-f7d9-274be5e3a9d3 (at 14079@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff884020d07c00, cur 1487880005 expire 1487879105 last 1487878653 <4>[1389217.171885] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1389220.248149] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1398136.576265] Lustre: atlas1-MDT0000: haven't heard from client 10721bf9-f34c-56bf-3be9-c5faf87e4da4 (at 1119@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0e1ba400, cur 1487893239 expire 1487892339 last 1487891887 <4>[1416028.551016] Lustre: atlas1-MDT0000: haven't heard from client 534a7a33-5bb0-9169-f487-64027b46ae9a (at 17664@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8815e1d97c00, cur 1487911131 expire 1487910231 last 1487909779 <4>[1432836.529914] Lustre: atlas1-MDT0000: haven't heard from client a1421eb4-9f37-2e19-5ecd-279417a080bd (at 1356@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883be4265400, cur 1487927939 expire 1487927039 last 1487926587 <4>[1436663.522491] Lustre: atlas1-MDT0000: haven't heard from client 21ae7933-eb6b-1b9d-ff54-fd770edf7e9e (at 5654@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883ed7704400, cur 1487931766 expire 1487930866 last 1487930414 <6>[1444301.099495] Lustre: atlas1-MDT0000: Connection restored to 19ec68a7-3341-e899-9a21-7185ac47b520 (at 0@lo) <6>[1444301.110843] Lustre: Skipped 15 previous similar messages <6>[1444301.117277] Lustre: client wants to enable acl, but mdt not! <4>[1444301.293711] Lustre: Mounted atlas1-client <4>[1446551.239078] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1446820.395970] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1446823.429873] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1446825.514831] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1446896.509082] Lustre: atlas1-MDT0000: haven't heard from client 7a3f3f26-fae9-b4dd-440e-095f16d7ba14 (at 15245@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0c9d9c00, cur 1487941999 expire 1487941099 last 1487940647 <4>[1447347.508356] Lustre: atlas1-MDT0000: haven't heard from client 2571d27d-c173-e2d2-c14f-b5922304e76d (at 13803@gni100) in 1136 seconds. I think it's dead, and I am evicting it. exp ffff883f10b76400, cur 1487942450 expire 1487941550 last 1487941314 <4>[1447347.533793] Lustre: Skipped 309 previous similar messages <6>[1448066.066886] Lustre: atlas1-MDT0000: Connection restored to b319406b-d04b-c6e0-747c-de41b4e0330b (at 10.39.232.89@o2ib6) <6>[1448655.384535] Lustre: atlas1-MDT0000: Connection restored to 6325695d-bcba-6b0a-c117-46793ad84772 (at 10.39.232.59@o2ib6) <4>[1449955.405364] Lustre: Unmounted atlas1-client <4>[1458797.242503] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1461223.245972] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1461226.358899] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1461228.719665] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1461716.487118] Lustre: atlas1-MDT0000: haven't heard from client 96af42fc-2437-710a-ade2-dc6c2d6c5788 (at 6099@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0fecc400, cur 1487956819 expire 1487955919 last 1487955467 <6>[1463371.295880] Lustre: atlas1-MDT0000: Connection restored to 5256c3ed-a954-77d4-aa60-766fb283bc77 (at 6270@gni100) <6>[1463371.930376] Lustre: atlas1-MDT0000: Connection restored to 0c830cd4-3343-99c4-5153-3ba9b8da826b (at 3196@gni100) <6>[1463371.942402] Lustre: Skipped 7 previous similar messages <6>[1463373.000250] Lustre: atlas1-MDT0000: Connection restored to 151119d8-709a-fcb6-cdeb-488a671f30f2 (at 13696@gni100) <6>[1463373.012378] Lustre: Skipped 21 previous similar messages <6>[1463375.021487] Lustre: atlas1-MDT0000: Connection restored to cc808f4f-ddf7-64d0-878d-36146418114c (at 4480@gni100) <6>[1463375.033524] Lustre: Skipped 26 previous similar messages <6>[1463379.056382] Lustre: atlas1-MDT0000: Connection restored to 137f6cc6-5bcc-23c0-6f18-c4daf4f6115b (at 13709@gni100) <6>[1463379.068497] Lustre: Skipped 62 previous similar messages <6>[1463387.093303] Lustre: atlas1-MDT0000: Connection restored to 03ad075c-9780-7f4d-1848-deab6928b6e1 (at 4433@gni100) <6>[1463387.105331] Lustre: Skipped 104 previous similar messages <6>[1463810.307684] Lustre: atlas1-MDT0000: Connection restored to 9ac239fe-0b1c-f6e6-bfe4-81bbc70443e5 (at 13938@gni100) <6>[1463810.319799] Lustre: Skipped 85 previous similar messages <4>[1464352.484923] Lustre: atlas1-MDT0000: haven't heard from client 0f959279-8f63-6f14-dda3-a5ef584ae638 (at 13609@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f10baf800, cur 1487959455 expire 1487958555 last 1487958103 <3>[1468566.566433] INFO: task mdt00_414:15873 blocked for more than 120 seconds. <3>[1468566.574484] Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <3>[1468566.581658] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. <6>[1468566.591061] mdt00_414 D 0000000000000001 0 15873 2 0x00000000 <4>[1468566.599233] ffff883f2a3ff7c0 0000000000000046 ffff88018fc76f00 0000000000000003 <4>[1468566.608294] ffff883f2a3ff740 ffffffff81063876 ffff883f2a3ff750 ffff8840265e9520 <4>[1468566.617337] ffff883f2a3ff750 ffffffff8105fa4d ffff883f2a3f9ad8 ffff883f2a3fffd8 <4>[1468566.626372] Call Trace: <4>[1468566.629528] [] ? enqueue_task+0x66/0x80 <4>[1468566.636087] [] ? check_preempt_curr+0x6d/0x90 <4>[1468566.643266] [] schedule_timeout+0x215/0x2e0 <4>[1468566.650215] [] ? autoremove_wake_function+0x16/0x40 <4>[1468566.657950] [] ? __wake_up_common+0x59/0x90 <4>[1468566.664903] [] wait_for_common+0x123/0x180 <4>[1468566.671762] [] ? default_wake_function+0x0/0x20 <4>[1468566.679105] [] ? __queue_work+0x41/0x50 <4>[1468566.685692] [] wait_for_completion+0x1d/0x20 <4>[1468566.692743] [] call_usermodehelper_exec+0x10c/0x130 <4>[1468566.700495] [] mdt_identity_do_upcall+0x13d/0x4b0 [mdt] <4>[1468566.708623] [] ? groups_free+0x54/0x60 <4>[1468566.715094] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[1468566.722873] [] upcall_cache_get_entry+0x1b7/0x880 [obdclass] <4>[1468566.731738] [] ? null_alloc_rs+0xcd/0x320 [ptlrpc] <4>[1468566.739393] [] mdt_identity_get+0x17/0x40 [mdt] <4>[1468566.746739] [] old_init_ucred_common+0x7d/0x2b0 [mdt] <4>[1468566.754674] [] old_init_ucred+0x123/0x200 [mdt] <4>[1468566.762023] [] mdt_init_ucred_intent_getattr+0x9d/0xe0 [mdt] <4>[1468566.770886] [] mdt_intent_getattr+0x1e1/0x470 [mdt] <4>[1468566.778674] [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] <4>[1468566.786611] [] mdt_intent_policy+0x4be/0xc70 [mdt] <4>[1468566.794298] [] ldlm_lock_enqueue+0x127/0x990 [ptlrpc] <4>[1468566.802269] [] ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] <4>[1468566.810843] [] tgt_enqueue+0x61/0x230 [ptlrpc] <4>[1468566.818214] [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] <4>[1468566.826376] [] ptlrpc_main+0xd21/0x1800 [ptlrpc] <4>[1468566.833834] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] <4>[1468566.841297] [] kthread+0x9e/0xc0 <4>[1468566.847180] [] child_rip+0xa/0x20 <4>[1468566.853158] [] ? kthread+0x0/0xc0 <4>[1468566.859128] [] ? child_rip+0x0/0x20 <3>[1468566.865362] INFO: task mdt01_376:16347 blocked for more than 120 seconds. <3>[1468566.873403] Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <3>[1468566.880538] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. <6>[1468566.889951] mdt01_376 D 0000000000000006 0 16347 2 0x00000000 <4>[1468566.898104] ffff883f24cf37c0 0000000000000046 ffff88143fb9b918 ffff88143fb9b970 <4>[1468566.907098] ffff883f24cf3870 ffff880a3a1a5a20 ffff883f24cf3740 ffffffff811d0cf7 <4>[1468566.916106] ffff882b2c9f8f68 0000000000001000 ffff883f24ce05f8 ffff883f24cf3fd8 <4>[1468566.925169] Call Trace: <4>[1468566.928324] [] ? __find_get_block+0x97/0xe0 <4>[1468566.935274] [] ? put_dec+0x10c/0x110 <4>[1468566.941542] [] schedule_timeout+0x215/0x2e0 <4>[1468566.948485] [] wait_for_common+0x123/0x180 <4>[1468566.955333] [] ? default_wake_function+0x0/0x20 <4>[1468566.962692] [] ? __queue_work+0x41/0x50 <4>[1468566.969252] [] wait_for_completion+0x1d/0x20 <4>[1468566.976298] [] call_usermodehelper_exec+0x10c/0x130 <4>[1468566.984046] [] mdt_identity_do_upcall+0x13d/0x4b0 [mdt] <4>[1468566.992163] [] ? __kmalloc+0x21c/0x230 <4>[1468566.998625] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[1468567.006385] [] upcall_cache_get_entry+0x1b7/0x880 [obdclass] <5>[1468567.008382] nfs: server 172.30.16.12 not responding, still trying <5>[1468567.008593] nfs: server 172.30.16.12 OK <4>[1468567.027202] [] ? lustre_pack_reply_v2+0x1eb/0x280 [ptlrpc] <4>[1468567.035859] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] <4>[1468567.043504] [] mdt_identity_get+0x17/0x40 [mdt] <4>[1468567.050852] [] old_init_ucred_common+0x7d/0x2b0 [mdt] <4>[1468567.058791] [] mdt_init_ucred_reint+0x173/0x210 [mdt] <4>[1468567.066730] [] mdt_reint_internal+0x258/0x9f0 [mdt] <4>[1468567.074508] [] mdt_intent_reint+0x1f6/0x440 [mdt] <4>[1468567.082058] [] mdt_intent_policy+0x4be/0xc70 [mdt] <4>[1468567.089757] [] ldlm_lock_enqueue+0x127/0x990 [ptlrpc] <4>[1468567.097711] [] ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] <4>[1468567.106273] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] <4>[1468567.114267] [] tgt_enqueue+0x61/0x230 [ptlrpc] <4>[1468567.121548] [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] <4>[1468567.129733] [] ptlrpc_main+0xd21/0x1800 [ptlrpc] <4>[1468567.137204] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] <4>[1468567.144651] [] kthread+0x9e/0xc0 <4>[1468567.150543] [] child_rip+0xa/0x20 <4>[1468567.156524] [] ? kthread+0x0/0xc0 <4>[1468567.162503] [] ? child_rip+0x0/0x20 <3>[1468567.168678] INFO: task mdt01_426:16397 blocked for more than 120 seconds. <3>[1468567.176700] Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <3>[1468567.183844] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. <6>[1468567.193248] mdt01_426 D 0000000000000005 0 16397 2 0x00000000 <4>[1468567.201404] ffff883f245a77c0 0000000000000046 ffff883f245a7780 ffffffffa0e2b195 <4>[1468567.210378] 0000000000001000 ffff8807847d30c8 ffff883f245a7740 ffffffff811d0cf7 <4>[1468567.219390] ffff883f245a7800 0000000000001000 ffff883f2451d068 ffff883f245a7fd8 <4>[1468567.228402] Call Trace: <4>[1468567.231605] [] ? __ldiskfs_get_inode_loc+0xf5/0x3b0 [ldiskfs] <4>[1468567.240524] [] ? __find_get_block+0x97/0xe0 <4>[1468567.247520] [] ? put_dec+0x10c/0x110 <4>[1468567.253796] [] schedule_timeout+0x215/0x2e0 <4>[1468567.260755] [] wait_for_common+0x123/0x180 <4>[1468567.267657] [] ? default_wake_function+0x0/0x20 <4>[1468567.275037] [] ? __queue_work+0x41/0x50 <4>[1468567.281608] [] wait_for_completion+0x1d/0x20 <4>[1468567.288675] [] call_usermodehelper_exec+0x10c/0x130 <4>[1468567.296450] [] mdt_identity_do_upcall+0x13d/0x4b0 [mdt] <4>[1468567.304580] [] ? groups_free+0x54/0x60 <4>[1468567.311050] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[1468567.318792] [] upcall_cache_get_entry+0x1b7/0x880 [obdclass] <4>[1468567.327662] [] ? null_alloc_rs+0xcd/0x320 [ptlrpc] <4>[1468567.335316] [] mdt_identity_get+0x17/0x40 [mdt] <4>[1468567.342696] [] old_init_ucred_common+0x7d/0x2b0 [mdt] <4>[1468567.350633] [] old_init_ucred+0x123/0x200 [mdt] <4>[1468567.357985] [] mdt_init_ucred_intent_getattr+0x9d/0xe0 [mdt] <4>[1468567.366832] [] mdt_intent_getattr+0x1e1/0x470 [mdt] <4>[1468567.374591] [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] <4>[1468567.382537] [] mdt_intent_policy+0x4be/0xc70 [mdt] <4>[1468567.390206] [] ldlm_lock_enqueue+0x127/0x990 [ptlrpc] <4>[1468567.398172] [] ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] <4>[1468567.406749] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] <4>[1468567.414723] [] tgt_enqueue+0x61/0x230 [ptlrpc] <4>[1468567.422015] [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] <4>[1468567.430216] [] ptlrpc_main+0xd21/0x1800 [ptlrpc] <4>[1468567.437702] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] <4>[1468567.445142] [] kthread+0x9e/0xc0 <4>[1468567.451032] [] child_rip+0xa/0x20 <4>[1468567.457018] [] ? kthread+0x0/0xc0 <4>[1468567.463005] [] ? child_rip+0x0/0x20 <5>[1468567.478381] nfs: server 172.30.16.12 not responding, still trying <5>[1468567.485634] nfs: server 172.30.16.12 not responding, still trying <5>[1468567.486240] nfs: server 172.30.16.12 OK <5>[1468567.505244] nfs: server 172.30.16.12 OK <5>[1468568.412380] nfs: server 172.30.16.12 not responding, still trying <5>[1468568.421419] nfs: server 172.30.16.12 OK <5>[1468570.661381] nfs: server 172.30.16.12 not responding, still trying <5>[1468570.668863] nfs: server 172.30.16.12 OK <5>[1468573.871375] nfs: server 172.30.16.12 not responding, still trying <5>[1468573.878820] nfs: server 172.30.16.12 OK <5>[1468573.906371] nfs: server 172.30.16.12 not responding, still trying <5>[1468573.913898] nfs: server 172.30.16.12 OK <5>[1468576.217372] nfs: server 172.30.16.12 not responding, still trying <5>[1468576.224820] nfs: server 172.30.16.12 OK <5>[1468578.136368] nfs: server 172.30.16.12 not responding, still trying <5>[1468578.144498] nfs: server 172.30.16.12 OK <5>[1468581.478363] nfs: server 172.30.16.12 not responding, still trying <5>[1468581.485784] nfs: server 172.30.16.12 OK <5>[1468598.552340] nfs: server 172.30.16.12 not responding, still trying <5>[1468598.559817] nfs: server 172.30.16.12 OK <5>[1468601.954342] nfs: server 172.30.16.12 not responding, still trying <5>[1468601.961778] nfs: server 172.30.16.12 OK <4>[1470422.241387] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1475619.296792] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1475622.433856] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1475624.593543] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1478867.462513] Lustre: atlas1-MDT0000: haven't heard from client cba0acfc-8bcd-49cc-9c83-138c8d561252 (at 8900@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f12379400, cur 1487973970 expire 1487973070 last 1487972618 <4>[1482062.240782] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1493522.239220] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1505204.233632] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1505365.425202] Lustre: atlas1-MDT0000: haven't heard from client 91744af7-1c2e-9a5c-ea4d-0fbecda2966a (at 17379@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0fe56000, cur 1488000468 expire 1487999568 last 1487999116 <4>[1516676.230215] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1516992.408670] Lustre: atlas1-MDT0000: haven't heard from client 447a3035-c9fc-55f6-9d0d-6a9e09c3d58e (at 16950@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff882ea24b5000, cur 1488012095 expire 1488011195 last 1488010743 <4>[1528361.225578] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1528970.395609] Lustre: atlas1-MDT0000: haven't heard from client 17e0ec25-6603-3e0e-a9f0-c3e77219b826 (at 8622@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f11f1dc00, cur 1488024073 expire 1488023173 last 1488022721 <4>[1529421.391286] Lustre: atlas1-MDT0000: haven't heard from client 93c43967-50a9-eb89-c1c4-d7f422c280b6 (at 18979@gni100) in 1115 seconds. I think it's dead, and I am evicting it. exp ffff883f0bfe5000, cur 1488024524 expire 1488023624 last 1488023409 <4>[1533226.915198] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1533230.116043] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1533232.246043] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1540079.221632] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <6>[1541432.305554] Lustre: atlas1-MDT0000: Connection restored to ce742519-4723-712a-be78-60e3cd9f9176 (at 10.39.232.64@o2ib6) <6>[1541432.318265] Lustre: Skipped 117 previous similar messages <6>[1541436.695907] Lustre: atlas1-MDT0000: Connection restored to 637b8411-2ef3-97c8-b22a-82548872fc6a (at 10.39.232.68@o2ib6) <6>[1541436.708612] Lustre: Skipped 3 previous similar messages <4>[1547620.484687] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1547623.510710] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1547625.593312] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1552121.218031] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1562020.608927] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1562023.712393] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1562025.900852] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1563921.227427] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <3>[1564883.738879] LustreError: 68491:0:(ldlm_lockd.c:342:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 17017@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff8803f4157b40/0xd94b67618b8e277d lrc: 4/0,0 mode: PR/PR res: [0x2002e1c31:0x1d947:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x60200400000020 nid: 17017@gni100 remote: 0xb113e561dbd94696 expref: 22 pid: 15606 timeout: 5859551115 lvb_type: 0 <3>[1564883.782880] LustreError: 15614:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) ### client (nid 17017@gni100) failed to reply to blocking AST (req status 0 rc -5), evict it ns: mdt-atlas1-MDT0000_UUID lock: ffff8803f4157b40/0xd94b67618b8e277d lrc: 4/0,0 mode: PR/PR res: [0x2002e1c31:0x1d947:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x60200400000020 nid: 17017@gni100 remote: 0xb113e561dbd94696 expref: 22 pid: 15606 timeout: 5859551115 lvb_type: 0 <3>[1564883.828236] LustreError: 138-a: atlas1-MDT0000: A client on nid 17017@gni100 was evicted due to a lock blocking callback time out: rc -5 <3>[1568794.733384] LustreError: 16135:0:(ldlm_lockd.c:342:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 18261@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff8801aab77480/0xd94b6761bd830890 lrc: 4/0,0 mode: PR/PR res: [0x20021dfd1:0x11842:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x60200400000020 nid: 18261@gni100 remote: 0x43a7eb0ed75c8cc expref: 20 pid: 15603 timeout: 5863462217 lvb_type: 0 <3>[1568794.781816] LustreError: 16415:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) ### client (nid 18261@gni100) failed to reply to blocking AST (req status 0 rc -5), evict it ns: mdt-atlas1-MDT0000_UUID lock: ffff8801aab77480/0xd94b6761bd830890 lrc: 4/0,0 mode: PR/PR res: [0x20021dfd1:0x11842:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x60200400000020 nid: 18261@gni100 remote: 0x43a7eb0ed75c8cc expref: 20 pid: 15603 timeout: 5863462217 lvb_type: 0 <3>[1568794.827152] LustreError: 138-a: atlas1-MDT0000: A client on nid 18261@gni100 was evicted due to a lock blocking callback time out: rc -5 <4>[1569388.056907] Lustre: atlas1-MDT0000: Client 7231b878-41b9-852a-8add-292f258ba242 (at 3790@gni100) reconnecting <6>[1569388.068707] Lustre: atlas1-MDT0000: Connection restored to 7231b878-41b9-852a-8add-292f258ba242 (at 3790@gni100) <4>[1569441.025347] Lustre: atlas1-MDT0000: Client d7d8fb65-d8cb-a15a-3da0-131c8bf2e492 (at 934@gni100) reconnecting <6>[1569441.077935] Lustre: atlas1-MDT0000: Connection restored to d7d8fb65-d8cb-a15a-3da0-131c8bf2e492 (at 934@gni100) <4>[1569514.918499] Lustre: atlas1-MDT0000: Client 4f305a8c-4967-b18f-9d4e-0404cbbbddb0 (at 16973@gni100) reconnecting <6>[1569515.465148] Lustre: atlas1-MDT0000: Connection restored to f45adde5-5c10-3f02-266d-f241a1799d2a (at 18358@gni100) <4>[1569539.270155] Lustre: atlas1-MDT0000: Client 1bde89a8-239a-7d87-3d24-89127efb7e68 (at 3782@gni100) reconnecting <4>[1569539.281901] Lustre: Skipped 2 previous similar messages <6>[1569539.461391] Lustre: atlas1-MDT0000: Connection restored to 1bde89a8-239a-7d87-3d24-89127efb7e68 (at 3782@gni100) <6>[1569539.473410] Lustre: Skipped 2 previous similar messages <4>[1569574.279003] Lustre: atlas1-MDT0000: Client 75b93fe3-e129-a472-824c-e94f0e7eb1d6 (at 18413@gni100) reconnecting <6>[1569574.645562] Lustre: atlas1-MDT0000: Connection restored to 75b93fe3-e129-a472-824c-e94f0e7eb1d6 (at 18413@gni100) <4>[1569616.135326] Lustre: atlas1-MDT0000: Client f0143492-d529-8c45-7ccb-ea1baeba1080 (at 18355@gni100) reconnecting <6>[1569616.174717] Lustre: atlas1-MDT0000: Connection restored to f0143492-d529-8c45-7ccb-ea1baeba1080 (at 18355@gni100) <4>[1569708.021641] Lustre: atlas1-MDT0000: Client b06c7663-062b-81a6-09d7-be268b39e13d (at 18359@gni100) reconnecting <6>[1569709.231810] Lustre: atlas1-MDT0000: Connection restored to b06c7663-062b-81a6-09d7-be268b39e13d (at 18359@gni100) <4>[1569797.542042] Lustre: atlas1-MDT0000: Client 3973e22a-6c25-db2f-e722-ec14a16226cc (at 16919@gni100) reconnecting <4>[1569797.553866] Lustre: Skipped 2 previous similar messages <6>[1569797.608340] Lustre: atlas1-MDT0000: Connection restored to 3973e22a-6c25-db2f-e722-ec14a16226cc (at 16919@gni100) <6>[1569797.620453] Lustre: Skipped 2 previous similar messages <4>[1570110.357309] Lustre: atlas1-MDT0000: Client 4a06f667-bc21-b7f0-7e6b-ebca857fb536 (at 16918@gni100) reconnecting <6>[1570110.615919] Lustre: atlas1-MDT0000: Connection restored to 4a06f667-bc21-b7f0-7e6b-ebca857fb536 (at 16918@gni100) <4>[1570311.314970] Lustre: atlas1-MDT0000: Client 306d7c55-8fc6-f1fd-1bf1-b0ba76cc339a (at 16912@gni100) reconnecting <4>[1570311.326801] Lustre: Skipped 1 previous similar message <6>[1570414.562355] Lustre: atlas1-MDT0000: Connection restored to 6a07ded9-3a34-5575-66ac-533041650cfe (at 911@gni100) <6>[1570414.574272] Lustre: Skipped 4 previous similar messages <4>[1570573.196968] Lustre: atlas1-MDT0000: Client 1bde89a8-239a-7d87-3d24-89127efb7e68 (at 3782@gni100) reconnecting <4>[1570573.208695] Lustre: Skipped 4 previous similar messages <4>[1571087.370823] Lustre: atlas1-MDT0000: Client 38a3207d-8e1d-335b-c639-66955a91567e (at 16917@gni100) reconnecting <4>[1571087.382644] Lustre: Skipped 7 previous similar messages <6>[1571087.391232] Lustre: atlas1-MDT0000: Connection restored to 38a3207d-8e1d-335b-c639-66955a91567e (at 16917@gni100) <6>[1571087.403365] Lustre: Skipped 9 previous similar messages <4>[1571964.277325] Lustre: atlas1-MDT0000: Client 75b93fe3-e129-a472-824c-e94f0e7eb1d6 (at 18413@gni100) reconnecting <4>[1571964.289145] Lustre: Skipped 9 previous similar messages <6>[1571964.296004] Lustre: atlas1-MDT0000: Connection restored to 75b93fe3-e129-a472-824c-e94f0e7eb1d6 (at 18413@gni100) <6>[1571964.308123] Lustre: Skipped 9 previous similar messages <4>[1572953.296319] Lustre: atlas1-MDT0000: Client 4a06f667-bc21-b7f0-7e6b-ebca857fb536 (at 16918@gni100) reconnecting <4>[1572953.308152] Lustre: Skipped 10 previous similar messages <6>[1572953.314518] Lustre: atlas1-MDT0000: Connection restored to 4a06f667-bc21-b7f0-7e6b-ebca857fb536 (at 16918@gni100) <6>[1572953.326657] Lustre: Skipped 10 previous similar messages <3>[1574105.725756] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 16898@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff881e29be12c0/0xd94b6761d25d9894 lrc: 4/0,0 mode: PR/PR res: [0x200395c07:0x2347:0x0].0x0 bits 0x2 rrc: 18 type: IBT flags: 0x60200400000020 nid: 16898@gni100 remote: 0xebab4dcf1f288685 expref: 15 pid: 16263 timeout: 5868773988 lvb_type: 0 <6>[1574212.452385] Lustre: atlas1-MDT0000: Connection restored to af542229-bb02-647d-08f8-233d558015a5 (at 16898@gni100) <4>[1574325.716432] Lustre: 15782:0:(service.c:1336:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[1574325.716433] req@ffff881bfa3a56c0 x1558697830915276/t0(0) o101->032237fc-a54c-19cc-319d-39a0c6c25b9e@18344@gni100:673/0 lens 704/3384 e 12 to 0 dl 1488069433 ref 2 fl Interpret:/2/0 rc 0/0 <3>[1574480.713195] LustreError: 16399:0:(ldlm_request.c:106:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1488068833, 750s ago); not entering recovery in server code, just going back to sleep ns: mdt-atlas1-MDT0000_UUID lock: ffff880d46d61bc0/0xd94b6761d25dbc90 lrc: 3/1,0 mode: --/PR res: [0x20037a539:0x4db5:0x0].0xdf06b5ba bits 0x2 rrc: 47 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 16399 timeout: 0 lvb_type: 0 <3>[1574480.759589] LustreError: 16399:0:(ldlm_request.c:106:ldlm_expired_completion_wait()) Skipped 42 previous similar messages <4>[1574580.748099] Lustre: atlas1-MDT0000: Client d96a4709-c0e1-016e-6a48-c2d9d9732a09 (at 92@gni100) reconnecting <6>[1574580.756283] Lustre: atlas1-MDT0000: Connection restored to 4686dd56-4c5b-0bd7-0d75-c44b794c8511 (at 18524@gni100) <4>[1574580.771822] Lustre: Skipped 26 previous similar messages <3>[1574630.724981] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) ### lock callback timer expired after 900s: evicting client at 16989@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff881d314f5580/0xd94b6761d25d8776 lrc: 4/0,0 mode: PR/PR res: [0x200395c07:0x2347:0x0].0x0 bits 0x2 rrc: 18 type: IBT flags: 0x60200400000020 nid: 16989@gni100 remote: 0x6d1b4689b7c65643 expref: 15 pid: 16032 timeout: 5869298988 lvb_type: 0 <4>[1574630.792093] Lustre: 16343:0:(service.c:2097:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:300s); client may timeout. req@ffff88283bb22c80 x1558697894525064/t651250688419(0) o36->d96a4709-c0e1-016e-6a48-c2d9d9732a09@92@gni100:673/0 lens 624/424 e 12 to 0 dl 1488069433 ref 1 fl Complete:/0/0 rc 0/0 <4>[1574630.825756] Lustre: 16343:0:(service.c:2097:ptlrpc_server_handle_request()) Skipped 46 previous similar messages <4>[1575628.326539] Lustre: 15471:0:(service.c:1336:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply <4>[1575628.326540] req@ffff8827ebb3a6c0 x1558697831192428/t0(0) o101->28e9cf10-49cb-359f-b2a6-733f095a3722@7@gni100:466/0 lens 704/3384 e 0 to 0 dl 1488070736 ref 2 fl Interpret:/2/0 rc 0/0 <4>[1575628.360058] Lustre: 15471:0:(service.c:1336:ptlrpc_at_send_early_reply()) Skipped 48 previous similar messages <3>[1575628.885538] LustreError: 16069:0:(ldlm_request.c:106:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1488069981, 750s ago); not entering recovery in server code, just going back to sleep ns: mdt-atlas1-MDT0000_UUID lock: ffff883e88118740/0xd94b6761d67225f7 lrc: 3/1,0 mode: --/PR res: [0x20037a539:0x4db5:0x0].0xcc1429ea bits 0x2 rrc: 4 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 16069 timeout: 0 lvb_type: 0 <3>[1575628.931774] LustreError: 16069:0:(ldlm_request.c:106:ldlm_expired_completion_wait()) Skipped 3 previous similar messages <3>[1575779.723347] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) ### lock callback timer expired after 901s: evicting client at 16987@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff8803e63736c0/0xd94b6761d67225c6 lrc: 4/0,0 mode: PR/PR res: [0x200395c07:0x2349:0x0].0x0 bits 0x2 rrc: 4 type: IBT flags: 0x60200400000020 nid: 16987@gni100 remote: 0x197be6bade9546b3 expref: 15 pid: 16112 timeout: 5870447162 lvb_type: 0 <3>[1575779.766655] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) Skipped 15 previous similar messages <4>[1575779.799367] Lustre: 16262:0:(service.c:2097:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:146s); client may timeout. req@ffff882831512380 x1558697894793224/t651251691262(0) o36->d96a4709-c0e1-016e-6a48-c2d9d9732a09@92@gni100:466/0 lens 616/424 e 0 to 0 dl 1488070736 ref 1 fl Complete:/0/0 rc 0/0 <4>[1575779.832949] Lustre: 16262:0:(service.c:2097:ptlrpc_server_handle_request()) Skipped 2 previous similar messages <6>[1575780.003168] Lustre: atlas1-MDT0000: Connection restored to 28e9cf10-49cb-359f-b2a6-733f095a3722 (at 7@gni100) <6>[1575780.014882] Lustre: Skipped 63 previous similar messages <4>[1577000.857599] Lustre: 16008:0:(service.c:1336:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[1577000.857600] req@ffff88283e29bcc0 x1558697895483540/t0(0) o36->d96a4709-c0e1-016e-6a48-c2d9d9732a09@92@gni100:328/0 lens 616/3128 e 12 to 0 dl 1488072108 ref 2 fl Interpret:/0/0 rc 0/0 <4>[1577000.890749] Lustre: 16008:0:(service.c:1336:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message <3>[1577155.859356] LustreError: 15527:0:(ldlm_request.c:106:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1488071508, 750s ago); not entering recovery in server code, just going back to sleep ns: mdt-atlas1-MDT0000_UUID lock: ffff881f71f521c0/0xd94b6761db6b33c1 lrc: 3/1,0 mode: --/PR res: [0x20037a539:0x4db5:0x0].0x70086d6b bits 0x2 rrc: 2 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 15527 timeout: 0 lvb_type: 0 <3>[1577155.905599] LustreError: 15527:0:(ldlm_request.c:106:ldlm_expired_completion_wait()) Skipped 1 previous similar message <4>[1577255.900322] Lustre: atlas1-MDT0000: Client d96a4709-c0e1-016e-6a48-c2d9d9732a09 (at 92@gni100) reconnecting <6>[1577255.904602] Lustre: atlas1-MDT0000: Connection restored to 937ff3b6-f696-a685-72f7-b00e392517c7 (at 93@gni100) <4>[1577255.923709] Lustre: Skipped 8 previous similar messages <3>[1577306.722150] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) ### lock callback timer expired after 901s: evicting client at 93@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff882cb008cb40/0xd94b6761db6b3247 lrc: 4/0,0 mode: PR/PR res: [0x200395c07:0x234f:0x0].0x0 bits 0x2 rrc: 2 type: IBT flags: 0x60200400000020 nid: 93@gni100 remote: 0x3a85f8da87ee2c8d expref: 14 pid: 15527 timeout: 5871974138 lvb_type: 0 <3>[1577306.765040] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) Skipped 2 previous similar messages <4>[1577306.797190] Lustre: 16272:0:(service.c:2097:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:301s); client may timeout. req@ffff88283e29bcc0 x1558697895483540/t651253126860(0) o36->d96a4709-c0e1-016e-6a48-c2d9d9732a09@92@gni100:328/0 lens 616/424 e 12 to 0 dl 1488072108 ref 1 fl Complete:/0/0 rc 0/0 <4>[1577306.830861] Lustre: 16272:0:(service.c:2097:ptlrpc_server_handle_request()) Skipped 1 previous similar message <6>[1577306.995933] Lustre: atlas1-MDT0000: Connection restored to 937ff3b6-f696-a685-72f7-b00e392517c7 (at 93@gni100) <6>[1577307.007777] Lustre: Skipped 1 previous similar message <4>[1578908.170533] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1580547.320932] Lustre: atlas1-MDT0000: haven't heard from client 317dee83-f6d9-42b8-597f-b7406798c574 (at 18394@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88120118a800, cur 1488075650 expire 1488074750 last 1488074298 <4>[1591006.166802] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1591297.306120] Lustre: atlas1-MDT0000: haven't heard from client b4d6d089-6d9e-5f08-4754-bb10706b5988 (at 5223@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0fc9dc00, cur 1488086400 expire 1488085500 last 1488085048 <4>[1602514.163014] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1603621.288257] Lustre: atlas1-MDT0000: haven't heard from client d0b73e80-fda9-fa94-c564-79fdfdcb563a (at 12541@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0c3f6c00, cur 1488098724 expire 1488097824 last 1488097372 <4>[1603621.313644] Lustre: Skipped 1 previous similar message <4>[1612992.277069] Lustre: atlas1-MDT0000: haven't heard from client 6f57dd4d-7144-f828-af0b-758d7987f073 (at 10610@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f127d9800, cur 1488108095 expire 1488107195 last 1488106743 <4>[1614463.163711] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <6>[1617100.299065] Lustre: atlas1-MDT0000: Connection restored to 10.36.226.117@o2ib (at 0@lo) <6>[1617100.308719] Lustre: client wants to enable acl, but mdt not! <4>[1617100.498415] Lustre: Mounted atlas1-client <4>[1617107.290807] telegraf: page allocation failure. order:1, mode:0x20 <4>[1617107.298034] Pid: 14452, comm: telegraf Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[1617107.307423] Call Trace: <4>[1617107.310571] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[1617107.318288] [] ? kmem_getpages+0x62/0x170 <4>[1617107.325034] [] ? fallback_alloc+0x1ba/0x270 <4>[1617107.331975] [] ? cache_grow+0x2cf/0x320 <4>[1617107.338527] [] ? ____cache_alloc_node+0x99/0x160 <4>[1617107.345979] [] ? lprocfs_stats_alloc_one+0x84/0x360 [obdclass] <4>[1617107.354979] [] ? __kmalloc+0x199/0x230 <4>[1617107.361453] [] ? lprocfs_stats_alloc_one+0x84/0x360 [obdclass] <4>[1617107.370477] [] ? lprocfs_counter_add+0x1a8/0x1c0 [obdclass] <4>[1617107.379217] [] ? after_reply+0x329/0xea0 [ptlrpc] <4>[1617107.386756] [] ? ptlrpc_check_set+0x1242/0x1d80 [ptlrpc] <4>[1617107.394980] [] ? ptlrpc_set_wait+0x1a0/0x960 [ptlrpc] <4>[1617107.402904] [] ? ll_statfs_internal+0x4c5/0xcb0 [lustre] <4>[1617107.411112] [] ? ll_statfs+0x95/0x190 [lustre] <4>[1617107.418342] [] ? statfs_by_dentry+0x74/0xa0 <4>[1617107.425283] [] ? vfs_statfs+0x1b/0xb0 <4>[1617107.431637] [] ? user_statfs+0x47/0xb0 <4>[1617107.438088] [] ? sys_statfs+0x2a/0x50 <4>[1617107.444444] [] ? system_call_fastpath+0x16/0x1b <6>[1617107.451773] Mem-Info: <4>[1617107.454724] Node 0 DMA per-cpu: <4>[1617107.458660] CPU 0: hi: 0, btch: 1 usd: 0 <4>[1617107.464432] CPU 1: hi: 0, btch: 1 usd: 0 <4>[1617107.470296] CPU 2: hi: 0, btch: 1 usd: 0 <4>[1617107.476066] CPU 3: hi: 0, btch: 1 usd: 0 <4>[1617107.481836] CPU 4: hi: 0, btch: 1 usd: 0 <4>[1617107.487605] CPU 5: hi: 0, btch: 1 usd: 0 <4>[1617107.493377] CPU 6: hi: 0, btch: 1 usd: 0 <4>[1617107.499143] CPU 7: hi: 0, btch: 1 usd: 0 <4>[1617107.504912] Node 0 DMA32 per-cpu: <4>[1617107.509039] CPU 0: hi: 186, btch: 31 usd: 0 <4>[1617107.514807] CPU 1: hi: 186, btch: 31 usd: 0 <4>[1617107.520578] CPU 2: hi: 186, btch: 31 usd: 0 <4>[1617107.526345] CPU 3: hi: 186, btch: 31 usd: 0 <4>[1617107.532114] CPU 4: hi: 186, btch: 31 usd: 0 <4>[1617107.537883] CPU 5: hi: 186, btch: 31 usd: 0 <4>[1617107.543654] CPU 6: hi: 186, btch: 31 usd: 0 <4>[1617107.549428] CPU 7: hi: 186, btch: 31 usd: 0 <4>[1617107.555195] Node 0 Normal per-cpu: <4>[1617107.559427] CPU 0: hi: 186, btch: 31 usd: 92 <4>[1617107.565197] CPU 1: hi: 186, btch: 31 usd: 68 <4>[1617107.570954] CPU 2: hi: 186, btch: 31 usd: 172 <4>[1617107.576723] CPU 3: hi: 186, btch: 31 usd: 138 <4>[1617107.582493] CPU 4: hi: 186, btch: 31 usd: 39 <4>[1617107.588262] CPU 5: hi: 186, btch: 31 usd: 56 <4>[1617107.599294] CPU 6: hi: 186, btch: 31 usd: 69 <4>[1617107.605063] CPU 7: hi: 186, btch: 31 usd: 42 <4>[1617107.610833] active_anon:637723 inactive_anon:121930 isolated_anon:0 <4>[1617107.610834] active_file:16799146 inactive_file:16800703 isolated_file:0 <4>[1617107.610834] unevictable:12414 dirty:6872 writeback:0 unstable:0 <4>[1617107.610835] free:189518 slab_reclaimable:29524494 slab_unreclaimable:1097929 <4>[1617107.610835] mapped:11193 shmem:177134 pagetables:2651 bounce:0 <4>[1617107.648943] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[1617107.690987] lowmem_reserve[]: 0 1880 258420 258420 <4>[1617107.696817] Node 0 DMA32 free:387068kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[1617107.739934] lowmem_reserve[]: 0 0 256540 256540 <4>[1617107.745465] Node 0 Normal free:344600kB min:67088kB low:83860kB high:100632kB active_anon:2550920kB inactive_anon:487724kB active_file:67201144kB inactive_file:67206792kB unevictable:49656kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:27524kB writeback:0kB mapped:44740kB shmem:708540kB slab_reclaimable:118101352kB slab_unreclaimable:4392044kB kernel_stack:75792kB pagetables:10560kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[1617107.795975] lowmem_reserve[]: 0 0 0 0 <4>[1617107.800551] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[1617107.813195] Node 0 DMA32: 7*4kB 6*8kB 5*16kB 9*32kB 7*64kB 5*128kB 6*256kB 10*512kB 8*1024kB 5*2048kB 88*4096kB = 387068kB <4>[1617107.826330] Node 0 Normal: 79839*4kB 166*8kB 513*16kB 158*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 333948kB <4>[1617107.840327] 33781766 total pagecache pages <4>[1617107.845319] 0 pages in swap cache <4>[1617107.849437] Swap cache stats: add 0, delete 0, find 0/0 <4>[1617107.855679] Free swap = 0kB <4>[1617107.859311] Total swap = 0kB <6>[1617108.258020] 67108863 pages RAM <6>[1617108.261846] 1015827 pages reserved <6>[1617108.266057] 33548412 pages shared <6>[1617108.270173] 32073558 pages non-shared <3>[1617108.274714] LustreError: 14452:0:(lprocfs_status.c:1045:lprocfs_stats_alloc_one()) LNET: out of memory at /tmp/rpmbuild-lustre-jsimmons-mRppNlWn/BUILD/lustre-2.8.0/lustre/obdclass/lprocfs_status.c:1045 (tried to alloc '(stats->ls_percpu[cpuid])' = 4352) <3>[1617108.300863] LustreError: 14452:0:(lprocfs_status.c:1045:lprocfs_stats_alloc_one()) LNET: 1302426760 total bytes allocated by lnet <4>[1619620.851603] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1619623.981382] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1619626.460256] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1620540.267540] Lustre: atlas1-MDT0000: haven't heard from client 7b4f3ca4-669d-0fdf-3d84-51c11e5d221c (at 812@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0d22ac00, cur 1488115643 expire 1488114743 last 1488114291 <4>[1622780.271646] Lustre: Unmounted atlas1-client <4>[1627454.157693] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1634024.542201] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1634027.812237] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1634029.859637] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1639063.155138] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <6>[1642813.707691] Lustre: atlas1-MDT0000: Connection restored to 3d57a70f-1c2f-76c0-fd76-519b15090f1c (at 10.39.232.87@o2ib6) <3>[1646193.627193] LustreError: 97292:0:(ldlm_lockd.c:342:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 17132@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff883aab0dc3c0/0xd94b676255ba23b4 lrc: 4/0,0 mode: PR/PR res: [0x20021dfd1:0x11842:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x60200400000020 nid: 17132@gni100 remote: 0x210b127723ace5ab expref: 22 pid: 15750 timeout: 5940861549 lvb_type: 0 <4>[1648423.473760] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1648426.514256] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1648428.940516] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1651033.159502] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1660584.906396] Lustre: 15767:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1488155120/real 1488155120] req@ffff882c23f55380 x1558715090973668/t0(0) o104->atlas1-MDT0000@4579@gni100:15/16 lens 296/224 e 0 to 1 dl 1488155687 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[1660612.819284] Lustre: 15948:0:(service.c:1336:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[1660612.819285] req@ffff882833917980 x1558697505835092/t0(0) o36->f8ab7c14-cb93-682b-78a5-975c4864d3f1@9306@gni100:135/0 lens 632/3128 e 12 to 0 dl 1488155720 ref 2 fl Interpret:/0/0 rc 0/0 <4>[1660868.488834] Lustre: atlas1-MDT0000: Client f8ab7c14-cb93-682b-78a5-975c4864d3f1 (at 9306@gni100) reconnecting <6>[1660868.500571] Lustre: atlas1-MDT0000: Connection restored to f8ab7c14-cb93-682b-78a5-975c4864d3f1 (at 9306@gni100) <4>[1661152.018617] Lustre: 15767:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1488155687/real 1488155687] req@ffff882c23f55380 x1558715090973668/t0(0) o104->atlas1-MDT0000@4579@gni100:15/16 lens 296/224 e 0 to 1 dl 1488156254 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 <4>[1661152.050026] Lustre: 15767:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 1 previous similar message <3>[1661152.061575] LustreError: 15767:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) ### client (nid 4579@gni100) failed to reply to blocking AST (req status 0 rc -110), evict it ns: mdt-atlas1-MDT0000_UUID lock: ffff8803cd696180/0xd94b676272e38fa2 lrc: 4/0,0 mode: PR/PR res: [0x200395d44:0xbf9b:0x0].0x0 bits 0x1b rrc: 3 type: IBT flags: 0x60200400000020 nid: 4579@gni100 remote: 0x9e8fe327f7b9d0ad expref: 23 pid: 15955 timeout: 5956153330 lvb_type: 0 <3>[1661152.106837] LustreError: 138-a: atlas1-MDT0000: A client on nid 4579@gni100 was evicted due to a lock blocking callback time out: rc -110 <4>[1661152.122363] Lustre: 15767:0:(service.c:2097:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:534s); client may timeout. req@ffff882833917980 x1558697505835092/t651375954846(0) o36->f8ab7c14-cb93-682b-78a5-975c4864d3f1@9306@gni100:135/0 lens 632/424 e 12 to 0 dl 1488155720 ref 1 fl Complete:/0/0 rc 0/0 <4>[1662706.155710] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1665725.203760] Lustre: atlas1-MDT0000: haven't heard from client 24cd3924-4666-e672-e3c9-a1ebd16a1b80 (at 191@gni4) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8826e6c13800, cur 1488160828 expire 1488159928 last 1488159476 <6>[1666369.535756] Lustre: atlas1-MDT0000: Connection restored to e0988e3d-a9e2-441f-7c5d-cca8e05ebb7d (at 17@gni4) <6>[1666372.936123] Lustre: atlas1-MDT0000: Connection restored to adf17aa3-aca8-71ac-4375-cca5d6d01a16 (at 129@gni4) <4>[1666775.202072] Lustre: atlas1-MDT0000: haven't heard from client 1aac0fb8-9f8f-1b5f-b11b-5230f3d8c9f4 (at 13958@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f1242bc00, cur 1488161878 expire 1488160978 last 1488160526 <4>[1666775.227458] Lustre: Skipped 167 previous similar messages <6>[1666801.771431] Lustre: atlas1-MDT0000: Connection restored to 1629421f-35e7-9355-900c-9ea9b3789ed3 (at 134@gni4) <6>[1666803.782419] Lustre: atlas1-MDT0000: Connection restored to c1344b89-3270-dc17-1a77-f6834f73e6c3 (at 137@gni4) <6>[1666803.794156] Lustre: Skipped 7 previous similar messages <6>[1666807.830731] Lustre: atlas1-MDT0000: Connection restored to b3a95257-421f-4d63-e767-23c2ebf2c95f (at 167@gni4) <6>[1666807.842448] Lustre: Skipped 119 previous similar messages <4>[1674028.153822] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1685639.150085] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <3>[1686408.950112] LustreError: 15874:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a2cad:0x9a66:0x0]: rc = -2 <4>[1696831.146484] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1706026.109257] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1706029.310156] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1706031.397882] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1708279.145200] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1709015.144257] Lustre: atlas1-MDT0000: haven't heard from client 1889f8a9-eed6-7625-8800-c5f384cfb6c5 (at 17063@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0c664800, cur 1488204118 expire 1488203218 last 1488202766 <4>[1709015.169639] Lustre: Skipped 2 previous similar messages <3>[1711758.772044] LustreError: 15783:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a10a1:0x92:0x0]: rc = -2 <4>[1716396.134445] Lustre: atlas1-MDT0000: haven't heard from client df4942e7-c422-9b9f-d7a4-ca527174b9b9 (at 1426@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883c14c43800, cur 1488211499 expire 1488210599 last 1488210147 <4>[1716396.159747] Lustre: Skipped 30 previous similar messages <4>[1719643.305380] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1720422.337233] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1720425.303707] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1720427.770082] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <6>[1720761.579930] Lustre: atlas1-MDT0000: Connection restored to d0c9774d-c245-e7a0-6746-fc90902e1f2d (at 10144@gni100) <6>[1720761.592066] Lustre: Skipped 39 previous similar messages <6>[1720762.653251] Lustre: atlas1-MDT0000: Connection restored to 5146577d-380f-757d-377c-86cbfc58523f (at 17584@gni100) <6>[1720762.665368] Lustre: Skipped 7 previous similar messages <6>[1720765.021808] Lustre: atlas1-MDT0000: Connection restored to 9bd342d5-34c9-2662-e357-d388ae53bfd0 (at 8622@gni100) <6>[1720765.033854] Lustre: Skipped 5 previous similar messages <6>[1720769.599166] Lustre: atlas1-MDT0000: Connection restored to 5ec9ae98-2aa6-bb2e-0b61-a8162dbfd0cf (at 236@gni100) <6>[1720769.611108] Lustre: Skipped 19 previous similar messages <6>[1720777.629832] Lustre: atlas1-MDT0000: Connection restored to f24d7d90-2557-d862-bf1b-5c766c5e2eff (at 18873@gni100) <6>[1720777.641973] Lustre: Skipped 39 previous similar messages <6>[1721151.937233] Lustre: atlas1-MDT0000: Connection restored to 2e040566-a757-abdb-5448-25db632bebb5 (at 18261@gni100) <6>[1721151.954514] Lustre: Skipped 24 previous similar messages <4>[1731188.128050] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1734823.581999] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1734826.722256] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1734829.080117] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <6>[1735900.621338] Lustre: atlas1-MDT0000: Connection restored to 0e195a23-3d44-06e8-4ca7-935ac02b9871 (at 0@lo) <6>[1735900.632701] Lustre: Skipped 22 previous similar messages <6>[1735900.639178] Lustre: client wants to enable acl, but mdt not! <4>[1735900.823821] Lustre: Mounted atlas1-client <4>[1741361.108799] Lustre: Unmounted atlas1-client <3>[1741547.343718] LustreError: 15768:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x200378313:0xb6c:0x0]: rc = -2 <4>[1744369.271966] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <3>[1744646.858836] LustreError: 15997:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x200382362:0x500d:0x0]: rc = -2 <3>[1744658.564497] LustreError: 15997:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20038213b:0xc2e:0x0]: rc = -2 <4>[1754355.078690] Lustre: atlas1-MDT0000: haven't heard from client 293fc0d3-1bbf-4cbc-5031-da0758f7ff92 (at 12895@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0ee00400, cur 1488249458 expire 1488248558 last 1488248106 <4>[1754355.104076] Lustre: Skipped 44 previous similar messages <4>[1755294.080275] Lustre: atlas1-MDT0000: haven't heard from client bfaddf25-e387-67c1-3f07-8cd0538de29b (at 8993@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f101c5800, cur 1488250397 expire 1488249497 last 1488249045 <4>[1756415.256403] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <3>[1763401.463147] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 14930@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff883dad030500/0xd94b6765e04489a5 lrc: 4/0,0 mode: PR/PR res: [0x20039add7:0xe5b2:0x0].0x0 bits 0x1b rrc: 3 type: IBT flags: 0x60200400000020 nid: 14930@gni100 remote: 0xd3d994be0562e224 expref: 23 pid: 15626 timeout: 6058069600 lvb_type: 0 <4>[1765179.063800] Lustre: atlas1-MDT0000: haven't heard from client d2718c4f-1434-2a21-b656-6aabe0eaa998 (at 18315@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883fa7da7c00, cur 1488260282 expire 1488259382 last 1488258930 <4>[1766666.062368] Lustre: atlas1-MDT0000: haven't heard from client 09e07aec-fcd4-93a0-20df-27336abb8402 (at 2158@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f12b91800, cur 1488261769 expire 1488260869 last 1488260417 <4>[1766666.087659] Lustre: Skipped 1 previous similar message <4>[1768144.238639] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1779873.223306] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1785193.035922] Lustre: atlas1-MDT0000: haven't heard from client 71e9ca33-b9b2-97c4-fde0-0abd457d5fa3 (at 10522@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f11623000, cur 1488280296 expire 1488279396 last 1488278944 <4>[1791745.319213] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f47a:0xd8e0:0x0] with flags 0x4a, rc = 0 <4>[1792421.458958] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1792424.420232] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1792426.429525] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1792714.028161] Lustre: atlas1-MDT0000: haven't heard from client 88ee2b38-a653-d696-42f7-80590e1a5505 (at 10146@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f129e3400, cur 1488287817 expire 1488286917 last 1488286465 <3>[1793189.685602] LustreError: 15957:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a338b:0x123f3:0x0]: rc = -2 <4>[1803570.338714] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f47a:0xd8e0:0x0] with flags 0x4a, rc = 0 <3>[1803762.406689] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 12768@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff882137678700/0xd94b67681aeacdd1 lrc: 4/0,0 mode: PR/PR res: [0x20039add5:0x1286f:0x0].0x0 bits 0x1b rrc: 3 type: IBT flags: 0x60200400000020 nid: 12768@gni100 remote: 0x98c26389f7c7c9f8 expref: 24 pid: 15840 timeout: 6098430312 lvb_type: 0 <3>[1803762.450181] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) Skipped 1 previous similar message <4>[1806820.502221] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1806826.163747] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1806830.686374] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <6>[1810872.942560] Lustre: atlas1-MDT0000: Connection restored to b650edd9-d4a2-1000-8370-e6bd05139217 (at 8282@gni100) <6>[1810873.563236] Lustre: atlas1-MDT0000: Connection restored to ee5012d3-f6d8-0197-9cfa-9d8bb44e7bb9 (at 10154@gni100) <6>[1810873.575360] Lustre: Skipped 4 previous similar messages <6>[1810874.829108] Lustre: atlas1-MDT0000: Connection restored to b3da69a2-3621-f384-069c-e9861788ea92 (at 8272@gni100) <6>[1810874.841132] Lustre: Skipped 5 previous similar messages <6>[1810876.921309] Lustre: atlas1-MDT0000: Connection restored to 9e51e477-95f9-172d-51f6-eaf93daa3fbd (at 12890@gni100) <6>[1810876.933431] Lustre: Skipped 28 previous similar messages <6>[1810880.977444] Lustre: atlas1-MDT0000: Connection restored to b66f8bb7-b961-a8e7-59f6-6a897e9c0a75 (at 11346@gni100) <6>[1810880.989564] Lustre: Skipped 37 previous similar messages <6>[1810889.048970] Lustre: atlas1-MDT0000: Connection restored to 76cc61dc-e7fb-8133-0beb-fd8e4f2ea275 (at 6409@gni100) <6>[1810889.060990] Lustre: Skipped 96 previous similar messages <6>[1811288.490301] Lustre: atlas1-MDT0000: Connection restored to cf81f4da-269c-ba78-41c9-65f1ddd884d1 (at 8608@gni100) <6>[1811288.502327] Lustre: Skipped 81 previous similar messages <6>[1811320.816516] Lustre: atlas1-MDT0000: Connection restored to 968af8e0-a637-04c3-37d1-daf246192fa1 (at 9812@gni100) <6>[1811320.828531] Lustre: Skipped 96 previous similar messages <6>[1813354.113204] Lustre: atlas1-MDT0000: Connection restored to e9411eee-aefa-9dcc-3989-736f27668f34 (at 17664@gni100) <6>[1813354.125345] Lustre: Skipped 3 previous similar messages <4>[1815555.172920] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <3>[1820105.515401] LustreError: 15718:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a34e9:0xed10:0x0]: rc = -2 <4>[1821217.109157] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1821220.291154] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1821222.338354] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <6>[1821245.615104] Lustre: atlas1-MDT0000: Connection restored to 8fca0f33-9907-2bc3-e816-8f958c601934 (at 10.39.232.104@o2ib6) <6>[1821245.627910] Lustre: Skipped 12 previous similar messages <6>[1821251.682996] Lustre: atlas1-MDT0000: Connection restored to 1f659bfd-9eb0-e41d-09e0-3679165bec04 (at 10.39.232.68@o2ib6) <6>[1821254.849519] Lustre: atlas1-MDT0000: Connection restored to fa7320d8-a034-7eb9-a71e-b02c9027ee5a (at 10.39.232.83@o2ib6) <6>[1821254.862241] Lustre: Skipped 1 previous similar message <4>[1821398.984434] Lustre: atlas1-MDT0000: haven't heard from client 170950b5-ddc5-5bc6-b0ab-bd930ac5b594 (at 2832@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f128c4c00, cur 1488316502 expire 1488315602 last 1488315150 <4>[1821399.009726] Lustre: Skipped 251 previous similar messages <6>[1821576.145099] Lustre: atlas1-MDT0000: Connection restored to b80b5ec1-d003-bcaa-ab62-c57ab76b5c1d (at 10.39.232.71@o2ib6) <4>[1821849.983834] Lustre: atlas1-MDT0000: haven't heard from client 4a95b4a6-37b7-590f-989a-9c67e20b9acf (at 10.39.232.71@o2ib6) in 990 seconds. I think it's dead, and I am evicting it. exp ffff883ed2526800, cur 1488316953 expire 1488316053 last 1488315963 <4>[1822300.983108] Lustre: atlas1-MDT0000: haven't heard from client 5916b1cd-c4f8-4025-2be8-697f5d6cf6e0 (at 10.39.232.57@o2ib6) in 1330 seconds. I think it's dead, and I am evicting it. exp ffff88205f62ac00, cur 1488317404 expire 1488316504 last 1488316074 <4>[1822301.009088] Lustre: Skipped 3 previous similar messages <4>[1827495.372827] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f47a:0xd8e0:0x0] with flags 0x4a, rc = 0 <4>[1839330.139334] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1851376.123484] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1856885.934843] Lustre: atlas1-MDT0000: haven't heard from client 07978cf5-02b1-85d6-27a8-1aec821662f5 (at 6708@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883ed004c400, cur 1488351989 expire 1488351089 last 1488350637 <4>[1863105.106804] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1868213.918646] Lustre: atlas1-MDT0000: haven't heard from client bf8e32f0-5462-29c7-af21-b2d2e164f841 (at 10447@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0f137400, cur 1488363317 expire 1488362417 last 1488361965 <3>[1873579.415329] LustreError: 15486:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a31b4:0x148aa:0x0]: rc = -2 <3>[1873579.430504] LustreError: 15486:0:(mdt_handler.c:893:mdt_getattr_internal()) Skipped 10 previous similar messages <3>[1873609.206304] LustreError: 15487:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a31b4:0x1693a:0x0]: rc = -2 <4>[1874827.973461] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <6>[1876351.388878] Lustre: atlas1-MDT0000: Connection restored to 526c88ad-c20d-575a-4fda-a8188a688d60 (at 0@lo) <6>[1876351.400386] Lustre: client wants to enable acl, but mdt not! <4>[1876351.588096] Lustre: Mounted atlas1-client <4>[1878824.674003] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1878828.318881] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1878830.490310] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1879245.903022] Lustre: atlas1-MDT0000: haven't heard from client 37c8848e-906e-c29a-63cf-070c699bf7e3 (at 12064@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f1188e000, cur 1488374349 expire 1488373449 last 1488372997 <4>[1881988.601266] Lustre: Unmounted atlas1-client <4>[1887828.473741] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f47a:0xd8e0:0x0] with flags 0x4a, rc = 0 <6>[1892872.466375] Lustre: atlas1-MDT0000: Connection restored to b0073342-a6e1-f7ef-0553-a8171ed5db0f (at 10.39.232.72@o2ib6) <6>[1892873.798237] Lustre: atlas1-MDT0000: Connection restored to cc2510c5-69a7-b5b9-9e54-e086d34c74bd (at 10.39.232.62@o2ib6) <6>[1892874.884988] Lustre: atlas1-MDT0000: Connection restored to 9c03449a-9ff9-8388-473e-16c13e7d8752 (at 10.39.232.53@o2ib6) <6>[1892874.902894] Lustre: Skipped 1 previous similar message <6>[1892877.072338] Lustre: atlas1-MDT0000: Connection restored to 583135a9-43a9-b7db-37ed-c5fe8390fc4a (at 10.39.232.95@o2ib6) <6>[1892891.667001] Lustre: atlas1-MDT0000: Connection restored to 064bace1-803e-e460-8094-24afd8de620b (at 10.39.232.86@o2ib6) <6>[1892891.679747] Lustre: Skipped 1 previous similar message <6>[1892900.028255] Lustre: atlas1-MDT0000: Connection restored to 2077a552-68a8-87ce-3dd4-bbd1a2bedb3b (at 10.39.232.94@o2ib6) <4>[1893217.170340] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1893220.309127] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <6>[1894048.939539] Lustre: atlas1-MDT0000: Connection restored to e60b8830-b906-c6fc-68b4-84896c73a6e5 (at 7898@gni100) <6>[1894048.939541] Lustre: atlas1-MDT0000: Connection restored to ecc7b86f-419d-c35f-72a4-82110b2f6d1f (at 7466@gni100) <6>[1894048.963594] Lustre: Skipped 1 previous similar message <6>[1894051.021723] Lustre: atlas1-MDT0000: Connection restored to 751c8dda-8685-9dd2-8901-ef9dc046561b (at 8994@gni100) <6>[1894051.033743] Lustre: Skipped 29 previous similar messages <6>[1894055.166502] Lustre: atlas1-MDT0000: Connection restored to 8eef6485-c549-58b2-b2bd-fab6d91f17e1 (at 10541@gni100) <6>[1894055.178627] Lustre: Skipped 45 previous similar messages <6>[1894063.355356] Lustre: atlas1-MDT0000: Connection restored to 68568bc8-13b4-91c9-0a32-0680c9c9bf64 (at 9476@gni100) <6>[1894063.367374] Lustre: Skipped 89 previous similar messages <4>[1894152.881917] Lustre: atlas1-MDT0000: haven't heard from client 8072a17e-9e8c-9695-9c46-fa77a1e54f83 (at 565@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0c01fc00, cur 1488389256 expire 1488388356 last 1488387904 <4>[1894152.907106] Lustre: Skipped 249 previous similar messages <6>[1894438.009341] Lustre: atlas1-MDT0000: Connection restored to 836b09fa-72c2-370e-718f-80eb38b943df (at 12502@gni100) <6>[1894438.021460] Lustre: Skipped 83 previous similar messages <6>[1896870.142436] Lustre: atlas1-MDT0000: Connection restored to 517979bf-fbab-0dae-3feb-ce53d1bb8b48 (at 10.36.247.131@o2ib) <6>[1896870.155142] Lustre: Skipped 94 previous similar messages <6>[1897297.971148] Lustre: atlas1-MDT0000: Connection restored to 5c7b9639-9684-98f7-2799-ef5fd8cb266d (at 10.36.247.132@o2ib) <4>[1897681.886102] Lustre: atlas1-MDT0000: haven't heard from client 215f38c6-16be-0f03-01bd-9b399c355961 (at 14035@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0d1c8c00, cur 1488392785 expire 1488391885 last 1488391433 <4>[1897681.911501] Lustre: Skipped 41 previous similar messages <4>[1899178.495845] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f47a:0xd8e0:0x0] with flags 0x4a, rc = 0 <6>[1899549.902607] Lustre: atlas1-MDT0000: Connection restored to 2553266c-09a3-c717-ac93-999cb494fc9d (at 234@gni100) <6>[1899549.914545] Lustre: Skipped 29 previous similar messages <6>[1899551.339683] Lustre: atlas1-MDT0000: Connection restored to 6b756468-102a-2673-eb05-86a958909104 (at 8110@gni100) <6>[1899551.351711] Lustre: Skipped 6 previous similar messages <6>[1899553.968950] Lustre: atlas1-MDT0000: Connection restored to e930bddc-ac42-845a-d7d6-76ca4f6bd297 (at 18000@gni100) <6>[1899553.981067] Lustre: Skipped 1 previous similar message <6>[1899558.355447] Lustre: atlas1-MDT0000: Connection restored to 9210fa73-9994-34fd-2879-53e2257934f6 (at 490@gni100) <6>[1899558.367387] Lustre: Skipped 10 previous similar messages <6>[1899566.447497] Lustre: atlas1-MDT0000: Connection restored to 926378fe-cb5c-b402-0a7a-cb1a7f7a12e2 (at 14091@gni100) <6>[1899566.459615] Lustre: Skipped 21 previous similar messages <4>[1899629.878527] Lustre: atlas1-MDT0000: haven't heard from client e2e2b98a-0e80-feee-d477-b8ec997f7a94 (at 2024@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f11963000, cur 1488394733 expire 1488393833 last 1488393381 <6>[1899916.534808] Lustre: atlas1-MDT0000: Connection restored to 0b3a3fd9-d3a6-84d3-e498-6a17d6d01ec0 (at 245@gni100) <6>[1899916.546734] Lustre: Skipped 13 previous similar messages <4>[1904329.874061] Lustre: atlas1-MDT0000: haven't heard from client 1c29074f-09a0-4c98-bbdc-a8505fe7aa8c (at 14031@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0e844400, cur 1488399433 expire 1488398533 last 1488398081 <3>[1906692.969969] LustreError: 15872:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a263b:0x11088:0x0]: rc = -2 <4>[1907337.888738] Lustre: atlas1-MDT0000: haven't heard from client a76ec9bf-26c6-70ff-01ff-df5e437c2e14 (at 17642@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0e896c00, cur 1488402441 expire 1488401541 last 1488401089 <4>[1907337.914159] Lustre: Skipped 1 previous similar message <4>[1907618.035636] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1907620.999154] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1907623.008777] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1911007.041920] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1916228.850671] Lustre: atlas1-MDT0000: haven't heard from client 97f8de93-aa4f-8885-3794-ef3a468b9407 (at 17836@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f11d0d400, cur 1488411332 expire 1488410432 last 1488409980 <4>[1922722.481848] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f47a:0xd8e0:0x0] with flags 0x4a, rc = 0 <4>[1926247.836459] Lustre: atlas1-MDT0000: haven't heard from client 083b25f2-5b31-c5b8-ad44-f6d2bff3f514 (at 17340@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0dd16c00, cur 1488421351 expire 1488420451 last 1488419999 <4>[1926247.861839] Lustre: Skipped 1 previous similar message <4>[1932248.829616] Lustre: atlas1-MDT0000: haven't heard from client 7a564b81-2aa7-3bf8-5051-ba4c3725b76b (at 6888@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880fc6c52c00, cur 1488427352 expire 1488426452 last 1488426000 <4>[1932248.854896] Lustre: Skipped 1 previous similar message <4>[1933831.010100] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1943478.812439] Lustre: atlas1-MDT0000: haven't heard from client d32ebb1a-0a24-37f6-3e05-44d960d781b4 (at 5803@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0a424000, cur 1488438582 expire 1488437682 last 1488437230 <4>[1944925.994189] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1948901.804827] Lustre: atlas1-MDT0000: haven't heard from client a5abcece-1d18-e0a7-a078-8a5226b82d74 (at 14984@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880a2df88c00, cur 1488444005 expire 1488443105 last 1488442653 <4>[1952503.799730] Lustre: atlas1-MDT0000: haven't heard from client 6b756468-102a-2673-eb05-86a958909104 (at 8110@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff882a6c794800, cur 1488447607 expire 1488446707 last 1488446255 <3>[1952811.526480] LustreError: 15699:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a33de:0x3c2e:0x0]: rc = -2 <4>[1956020.978814] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <3>[1960009.619403] LustreError: 15678:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a33de:0x71d4:0x0]: rc = -2 <3>[1960180.689815] LustreError: 15633:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a33de:0x88fe:0x0]: rc = -2 <4>[1965221.728252] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1965224.907548] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1965227.543707] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1965425.781143] Lustre: atlas1-MDT0000: haven't heard from client 5f9e9415-4b25-8ea7-f1fc-c60a856b182c (at 17464@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0d46cc00, cur 1488460529 expire 1488459629 last 1488459177 <4>[1966522.864554] Lustre: atlas1-MDT0000: haven't heard from client eb336bdc-886a-df28-26e0-1e65fede9ed4 (at 10198@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0dbdcc00, cur 1488461626 expire 1488460726 last 1488460274 <6>[1966581.985022] Lustre: atlas1-MDT0000: Connection restored to 761d8804-be4e-79d5-f6ce-c50a062e341d (at 10.39.232.92@o2ib6) <6>[1966581.997727] Lustre: Skipped 11 previous similar messages <4>[1967082.431820] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f47a:0xd8e0:0x0] with flags 0x4a, rc = 0 <4>[1974924.774973] Lustre: atlas1-MDT0000: haven't heard from client 43821906-5b27-e3d7-4867-e84975a4650d (at 11067@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0c160c00, cur 1488470028 expire 1488469128 last 1488468676 <4>[1974924.800427] Lustre: Skipped 246 previous similar messages <4>[1978265.762751] Lustre: atlas1-MDT0000: haven't heard from client 887aad67-6bf6-ad9c-853e-66c746e1f04f (at 1302@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0f29f000, cur 1488473369 expire 1488472469 last 1488472017 <4>[1978527.950376] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[1979622.248652] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1979625.295214] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1979627.806070] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <6>[1984633.341102] Lustre: atlas1-MDT0000: Connection restored to 20bfcc58-5a27-85d0-d679-da3d4579f84b (at 9720@gni100) <6>[1984633.880072] Lustre: atlas1-MDT0000: Connection restored to 36bd58c7-16bc-13d5-e233-782c7bd038ac (at 11254@gni100) <6>[1984633.892199] Lustre: Skipped 12 previous similar messages <6>[1984634.912706] Lustre: atlas1-MDT0000: Connection restored to 55f7adf5-44cf-dcba-57fc-9cf1e7cf27c8 (at 11782@gni100) <6>[1984634.924837] Lustre: Skipped 24 previous similar messages <6>[1984636.920662] Lustre: atlas1-MDT0000: Connection restored to bf734f9c-3bb7-58aa-d99f-de67e5d8ee20 (at 2158@gni100) <6>[1984636.932706] Lustre: Skipped 16 previous similar messages <6>[1984641.074840] Lustre: atlas1-MDT0000: Connection restored to 33cec5e1-ca93-932a-3385-d0a2c3cbc025 (at 9717@gni100) <6>[1984641.086863] Lustre: Skipped 58 previous similar messages <6>[1984649.094470] Lustre: atlas1-MDT0000: Connection restored to ecc86320-3932-44d3-8e32-56fe00a7644e (at 8231@gni100) <6>[1984649.106513] Lustre: Skipped 108 previous similar messages <6>[1985182.269695] Lustre: atlas1-MDT0000: Connection restored to 8fb5f711-b402-1151-bb41-fc108ca38edd (at 8719@gni100) <6>[1985182.281726] Lustre: Skipped 72 previous similar messages <6>[1985215.507496] Lustre: atlas1-MDT0000: Connection restored to 63b15fed-a68c-c691-7903-164ca0def94f (at 10247@gni100) <6>[1985215.519615] Lustre: Skipped 94 previous similar messages <3>[1989466.193495] LustreError: 16187:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20039c548:0xf96a:0x0]: rc = -2 <4>[1989763.418805] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f47a:0xd8e0:0x0] with flags 0x4a, rc = 0 <4>[1994027.999566] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1994030.999859] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[1994033.085913] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <6>[1996921.732203] Lustre: atlas1-MDT0000: Connection restored to 3db9b584-222e-fcc2-0ae4-8c051103fdef (at 10.39.232.51@o2ib6) <6>[1996921.744908] Lustre: Skipped 21 previous similar messages <6>[1996930.241744] Lustre: atlas1-MDT0000: Connection restored to 79ab8c70-ef46-2cf3-84b5-877bff4502e2 (at 10.39.232.71@o2ib6) <6>[1996930.254480] Lustre: Skipped 4 previous similar messages <6>[1996951.343057] Lustre: atlas1-MDT0000: Connection restored to 27146620-24ed-14eb-9bbb-c1134da37e5c (at 10.39.232.92@o2ib6) <6>[1996951.355764] Lustre: Skipped 5 previous similar messages <6>[1997246.987740] Lustre: atlas1-MDT0000: Connection restored to 0d0a28e5-1cf0-2907-9024-ea176695f98e (at 10.39.232.77@o2ib6) <6>[1997247.000436] Lustre: Skipped 2 previous similar messages <4>[1997884.750655] Lustre: atlas1-MDT0000: haven't heard from client dfbb65d1-102c-a282-0cf3-3309a0317a5c (at 10.39.232.67@o2ib6) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f15f2bc00, cur 1488492988 expire 1488492088 last 1488491636 <4>[1997884.776617] Lustre: Skipped 38 previous similar messages <4>[2001292.415821] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f47a:0xd8e0:0x0] with flags 0x4a, rc = 0 <4>[2007777.108903] Lustre: 14923:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1488502027/real 1488502027] req@ffff8801d7516980 x1558716256195872/t0(0) o6->atlas1-OST0026-osc-MDT0000@10.36.225.68@o2ib:28/4 lens 664/432 e 12 to 1 dl 1488502877 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[2007777.142123] Lustre: 14923:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 1 previous similar message <4>[2007777.153574] Lustre: atlas1-OST0026-osc-MDT0000: Connection to atlas1-OST0026 (at 10.36.225.68@o2ib) was lost; in progress operations using this service will wait for recovery to complete <4>[2007777.173078] Lustre: Skipped 1 previous similar message <6>[2007777.235911] Lustre: atlas1-OST0026-osc-MDT0000: Connection restored to 10.36.225.68@o2ib (at 10.36.225.68@o2ib) <4>[2007782.048893] Lustre: 14925:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1488502027/real 1488502027] req@ffff880bff7c9380 x1558716256195764/t0(0) o6->atlas1-OST0200-osc-MDT0000@10.36.225.110@o2ib:28/4 lens 664/432 e 12 to 1 dl 1488502877 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[2007782.082229] Lustre: atlas1-OST0200-osc-MDT0000: Connection to atlas1-OST0200 (at 10.36.225.110@o2ib) was lost; in progress operations using this service will wait for recovery to complete <4>[2007787.114218] Lustre: 14920:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1488502027/real 1488502027] req@ffff881a6d5a6680 x1558716256195852/t0(0) o6->atlas1-OST0267-osc-MDT0000@10.36.225.69@o2ib:28/4 lens 664/432 e 12 to 1 dl 1488502877 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[2007787.147415] Lustre: atlas1-OST0267-osc-MDT0000: Connection to atlas1-OST0267 (at 10.36.225.69@o2ib) was lost; in progress operations using this service will wait for recovery to complete <6>[2007787.220051] Lustre: atlas1-OST0267-osc-MDT0000: Connection restored to 10.36.225.69@o2ib (at 10.36.225.69@o2ib) <6>[2007787.231986] Lustre: Skipped 1 previous similar message <4>[2008268.730866] Lustre: atlas1-MDT0000: haven't heard from client 5b0a6e41-697d-244b-c91e-fcb5e9fc392e (at 8562@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880f47d41800, cur 1488503372 expire 1488502472 last 1488502020 <4>[2008268.756168] Lustre: Skipped 10 previous similar messages <4>[2010387.719143] Lustre: atlas1-MDT0000: haven't heard from client f15c8412-b902-07e8-6e87-10716602efc7 (at 1366@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0e798c00, cur 1488505491 expire 1488504591 last 1488504139 <4>[2013080.905759] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2013576.568969] kiblnd_sd_00_01: page allocation failure. order:1, mode:0x20 <4>[2013576.576880] Pid: 14828, comm: kiblnd_sd_00_01 Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[2013576.586945] Call Trace: <4>[2013576.590090] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[2013576.598726] [] ? select_idle_sibling+0x95/0x150 <4>[2013576.606056] [] ? kmem_getpages+0x62/0x170 <4>[2013576.612805] [] ? fallback_alloc+0x1ba/0x270 <4>[2013576.619747] [] ? cache_grow+0x2cf/0x320 <4>[2013576.626295] [] ? ____cache_alloc_node+0x99/0x160 <4>[2013576.633721] [] ? kmem_cache_alloc_node_trace+0x90/0x200 <4>[2013576.641815] [] ? __kmalloc_node+0x4d/0x60 <4>[2013576.648559] [] ? __alloc_skb+0x7a/0x190 <4>[2013576.655107] [] ? dev_alloc_skb+0x1d/0x40 <4>[2013576.661760] [] ? ipoib_alloc_rx_skb+0x3f/0x200 [ib_ipoib] <4>[2013576.670274] [] ? mlx4_ib_poll_cq+0x52a/0xd30 [mlx4_ib] <4>[2013576.678282] [] ? ipoib_ib_handle_rx_wc+0x8c/0x300 [ib_ipoib] <4>[2013576.687085] [] ? ipoib_poll+0x14b/0x180 [ib_ipoib] <4>[2013576.694704] [] ? net_rx_action+0x103/0x300 <4>[2013576.701543] [] ? __do_softirq+0xe5/0x230 <4>[2013576.708197] [] ? mlx4_msi_x_interrupt+0x14/0x20 [mlx4_core] <4>[2013576.716909] [] ? call_softirq+0x1c/0x30 <4>[2013576.723456] [] ? do_softirq+0x65/0xa0 <4>[2013576.729809] [] ? irq_exit+0x85/0x90 <4>[2013576.735968] [] ? do_IRQ+0x75/0xf0 <4>[2013576.741933] [] ? ret_from_intr+0x0/0x11 <4>[2013576.748478] [] ? cfs_percpt_unlock+0x53/0xb0 [libcfs] <4>[2013576.757318] [] ? lnet_finalize+0x2bf/0x730 [lnet] <4>[2013576.764838] [] ? kiblnd_tx_done+0x152/0x430 [ko2iblnd] <4>[2013576.772849] [] ? __wake_up+0x53/0x70 <4>[2013576.779108] [] ? kiblnd_scheduler+0xe4c/0x1160 [ko2iblnd] <4>[2013576.787625] [] ? default_wake_function+0x0/0x20 <4>[2013576.794944] [] ? kiblnd_scheduler+0x0/0x1160 [ko2iblnd] <4>[2013576.803046] [] ? kthread+0x9e/0xc0 <4>[2013576.809107] [] ? child_rip+0xa/0x20 <4>[2013576.815267] [] ? kthread+0x0/0xc0 <4>[2013576.821232] [] ? child_rip+0x0/0x20 <4>[2016684.709756] Lustre: atlas1-MDT0000: haven't heard from client a1eea2af-31e9-ad5f-9504-ca4a2139be13 (at 684@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff882bafebe800, cur 1488511788 expire 1488510888 last 1488510436 <6>[2023974.959141] Lustre: atlas1-MDT0000: Connection restored to d70690de-d75d-d1d3-c2d4-4f8469e9c3a7 (at 10.39.232.59@o2ib6) <4>[2024492.889906] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2035707.777025] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2047316.862910] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2048725.673448] Lustre: atlas1-MDT0000: haven't heard from client e4322ad5-fd11-b341-2eb5-045be54cb56a (at 17056@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0daa5400, cur 1488543829 expire 1488542929 last 1488542477 <6>[2049100.012315] Lustre: atlas1-MDT0000: Connection restored to 10.36.226.117@o2ib (at 0@lo) <6>[2049100.022060] Lustre: client wants to enable acl, but mdt not! <4>[2049100.208371] Lustre: Mounted atlas1-client <4>[2049106.681154] telegraf: page allocation failure. order:1, mode:0x20 <4>[2049106.688377] Pid: 23764, comm: telegraf Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[2049106.697762] Call Trace: <4>[2049106.700914] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[2049106.708637] [] ? number+0x1f0/0x320 <4>[2049106.714803] [] ? kmem_getpages+0x62/0x170 <4>[2049106.721558] [] ? fallback_alloc+0x1ba/0x270 <4>[2049106.728498] [] ? cache_grow+0x2cf/0x320 <4>[2049106.735058] [] ? ____cache_alloc_node+0x99/0x160 <4>[2049106.742504] [] ? lprocfs_stats_alloc_one+0x84/0x360 [obdclass] <4>[2049106.751514] [] ? __kmalloc+0x199/0x230 <4>[2049106.757990] [] ? lprocfs_stats_alloc_one+0x84/0x360 [obdclass] <4>[2049106.767020] [] ? lprocfs_counter_add+0x1a8/0x1c0 [obdclass] <4>[2049106.775767] [] ? ptl_send_rpc+0x5be/0xea0 [ptlrpc] <4>[2049106.783425] [] ? ptlrpc_send_new_req+0x50e/0x9b0 [ptlrpc] <4>[2049106.791976] [] ? ptlrpc_set_wait+0x6a6/0x960 [ptlrpc] <4>[2049106.799891] [] ? osc_statfs_async+0xcc/0x210 [osc] <4>[2049106.807525] [] ? lov_statfs_async+0x15b/0x670 [lov] <4>[2049106.815257] [] ? ll_statfs_internal+0x4c5/0xcb0 [lustre] <4>[2049106.823463] [] ? __kmalloc+0x21c/0x230 <4>[2049106.829941] [] ? lprocfs_stats_alloc_one+0x84/0x360 [obdclass] <4>[2049106.838968] [] ? lprocfs_counter_add+0x1a8/0x1c0 [obdclass] <4>[2049106.847699] [] ? ll_statfs+0x95/0x190 [lustre] <4>[2049106.854936] [] ? statfs_by_dentry+0x74/0xa0 <4>[2049106.861877] [] ? vfs_statfs+0x1b/0xb0 <4>[2049106.868239] [] ? user_statfs+0x47/0xb0 <4>[2049106.874684] [] ? sys_statfs+0x2a/0x50 <4>[2049106.881032] [] ? system_call_fastpath+0x16/0x1b <6>[2049106.888350] Mem-Info: <4>[2049106.891297] Node 0 DMA per-cpu: <4>[2049106.895239] CPU 0: hi: 0, btch: 1 usd: 0 <4>[2049106.901010] CPU 1: hi: 0, btch: 1 usd: 0 <4>[2049106.906780] CPU 2: hi: 0, btch: 1 usd: 0 <4>[2049106.912556] CPU 3: hi: 0, btch: 1 usd: 0 <4>[2049106.918329] CPU 4: hi: 0, btch: 1 usd: 0 <4>[2049106.924105] CPU 5: hi: 0, btch: 1 usd: 0 <4>[2049106.929876] CPU 6: hi: 0, btch: 1 usd: 0 <4>[2049106.935651] CPU 7: hi: 0, btch: 1 usd: 0 <4>[2049106.941425] Node 0 DMA32 per-cpu: <4>[2049106.945562] CPU 0: hi: 186, btch: 31 usd: 0 <4>[2049106.951336] CPU 1: hi: 186, btch: 31 usd: 0 <4>[2049106.957108] CPU 2: hi: 186, btch: 31 usd: 0 <4>[2049106.962880] CPU 3: hi: 186, btch: 31 usd: 0 <4>[2049106.968647] CPU 4: hi: 186, btch: 31 usd: 0 <4>[2049106.974421] CPU 5: hi: 186, btch: 31 usd: 0 <4>[2049106.980193] CPU 6: hi: 186, btch: 31 usd: 0 <4>[2049106.991231] CPU 7: hi: 186, btch: 31 usd: 0 <4>[2049106.997000] Node 0 Normal per-cpu: <4>[2049107.001234] CPU 0: hi: 186, btch: 31 usd: 36 <4>[2049107.007005] CPU 1: hi: 186, btch: 31 usd: 75 <4>[2049107.012777] CPU 2: hi: 186, btch: 31 usd: 169 <4>[2049107.018551] CPU 3: hi: 186, btch: 31 usd: 16 <4>[2049107.024321] CPU 4: hi: 186, btch: 31 usd: 0 <4>[2049107.030095] CPU 5: hi: 186, btch: 31 usd: 0 <4>[2049107.035868] CPU 6: hi: 186, btch: 31 usd: 0 <4>[2049107.041637] CPU 7: hi: 186, btch: 31 usd: 0 <4>[2049107.047413] active_anon:511663 inactive_anon:147122 isolated_anon:29 <4>[2049107.047413] active_file:17284535 inactive_file:17284656 isolated_file:0 <4>[2049107.047414] unevictable:8613 dirty:1831 writeback:0 unstable:0 <4>[2049107.047414] free:609021 slab_reclaimable:28581726 slab_unreclaimable:756019 <4>[2049107.047414] mapped:11372 shmem:204873 pagetables:2326 bounce:0 <4>[2049107.085428] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2049107.127486] lowmem_reserve[]: 0 1880 258420 258420 <4>[2049107.133313] Node 0 DMA32 free:387068kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2049107.176417] lowmem_reserve[]: 0 0 256540 256540 <4>[2049107.181967] Node 0 Normal free:2033272kB min:67088kB low:83860kB high:100632kB active_anon:2047236kB inactive_anon:588488kB active_file:69138232kB inactive_file:69138440kB unevictable:34452kB isolated(anon):0kB isolated(file):24kB present:262696960kB mlocked:0kB dirty:7324kB writeback:0kB mapped:45488kB shmem:819492kB slab_reclaimable:114326748kB slab_unreclaimable:3023992kB kernel_stack:75760kB pagetables:9496kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[2049107.232403] lowmem_reserve[]: 0 0 0 0 <4>[2049107.236970] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[2049107.249606] Node 0 DMA32: 7*4kB 6*8kB 5*16kB 9*32kB 7*64kB 5*128kB 6*256kB 10*512kB 8*1024kB 5*2048kB 88*4096kB = 387068kB <4>[2049107.262819] Node 0 Normal: 503937*4kB 794*8kB 407*16kB 119*32kB 3*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2032612kB <4>[2049107.277105] 34775193 total pagecache pages <4>[2049107.282098] 0 pages in swap cache <4>[2049107.286221] Swap cache stats: add 0, delete 0, find 0/0 <4>[2049107.292476] Free swap = 0kB <4>[2049107.296114] Total swap = 0kB <6>[2049107.734050] 67108863 pages RAM <6>[2049107.737879] 1015827 pages reserved <6>[2049107.742098] 27505260 pages shared <6>[2049107.746214] 37693184 pages non-shared <3>[2049107.750817] LustreError: 23764:0:(lprocfs_status.c:1045:lprocfs_stats_alloc_one()) LNET: out of memory at /tmp/rpmbuild-lustre-jsimmons-mRppNlWn/BUILD/lustre-2.8.0/lustre/obdclass/lprocfs_status.c:1045 (tried to alloc '(stats->ls_percpu[cpuid])' = 4352) <3>[2049107.777029] LustreError: 23764:0:(lprocfs_status.c:1045:lprocfs_stats_alloc_one()) LNET: 1299009768 total bytes allocated by lnet <4>[2050829.940498] swapper: page allocation failure. order:1, mode:0x20 <4>[2050829.947624] Pid: 0, comm: swapper Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[2050829.956596] Call Trace: <4>[2050829.959748] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[2050829.968375] [] ? kmem_getpages+0x62/0x170 <4>[2050829.975120] [] ? fallback_alloc+0x1ba/0x270 <4>[2050829.982050] [] ? cache_grow+0x2cf/0x320 <4>[2050829.988600] [] ? ____cache_alloc_node+0x99/0x160 <4>[2050829.996024] [] ? kmem_cache_alloc_node_trace+0x90/0x200 <4>[2050830.004127] [] ? __kmalloc_node+0x4d/0x60 <4>[2050830.010868] [] ? __alloc_skb+0x7a/0x190 <4>[2050830.017417] [] ? dev_alloc_skb+0x1d/0x40 <4>[2050830.024069] [] ? ipoib_alloc_rx_skb+0x3f/0x200 [ib_ipoib] <4>[2050830.032586] [] ? mlx4_ib_poll_cq+0x52a/0xd30 [mlx4_ib] <4>[2050830.040593] [] ? ipoib_ib_handle_rx_wc+0x8c/0x300 [ib_ipoib] <4>[2050830.049397] [] ? ipoib_poll+0x14b/0x180 [ib_ipoib] <4>[2050830.057017] [] ? net_rx_action+0x103/0x300 <4>[2050830.063858] [] ? hrtimer_get_next_event+0xc3/0x100 <4>[2050830.071467] [] ? __do_softirq+0xe5/0x230 <4>[2050830.078126] [] ? mlx4_msi_x_interrupt+0x14/0x20 [mlx4_core] <4>[2050830.086835] [] ? call_softirq+0x1c/0x30 <4>[2050830.093383] [] ? do_softirq+0x65/0xa0 <4>[2050830.099736] [] ? irq_exit+0x85/0x90 <4>[2050830.105896] [] ? do_IRQ+0x75/0xf0 <4>[2050830.111861] [] ? ret_from_intr+0x0/0x11 <4>[2050830.118410] [] ? intel_idle+0xfe/0x1b0 <4>[2050830.125560] [] ? intel_idle+0xe1/0x1b0 <4>[2050830.132016] [] ? cpuidle_idle_call+0x7a/0xe0 <4>[2050830.139054] [] ? cpu_idle+0xb6/0x110 <4>[2050830.145314] [] ? start_secondary+0x2c0/0x316 <4>[2051620.248568] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2051623.392883] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2051625.583008] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2054923.896445] Lustre: Unmounted atlas1-client <4>[2056901.656671] Lustre: atlas1-MDT0000: haven't heard from client 2e3803ec-6793-4783-8d05-cde21cd690b7 (at 2896@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0e787c00, cur 1488552005 expire 1488551105 last 1488550653 <4>[2056901.682005] Lustre: Skipped 32 previous similar messages <4>[2057352.656823] Lustre: atlas1-MDT0000: haven't heard from client 7477acde-211e-a432-d536-45bfe6c0c4b0 (at 6837@gni100) in 1163 seconds. I think it's dead, and I am evicting it. exp ffff8813300cac00, cur 1488552456 expire 1488551556 last 1488551293 <4>[2058789.342737] Lustre: atlas1-MDT0000: Client 89256a0d-c0b7-be9f-503d-3d0a81d1c177 (at 11359@gni100) reconnecting <6>[2058789.354585] Lustre: atlas1-MDT0000: Connection restored to a42b866a-7065-fb09-1b27-a9005552035c (at 11359@gni100) <4>[2058805.578407] Lustre: atlas1-MDT0000: Client 4e662dec-f2ca-3ff6-894a-e4154254c4ca (at 11353@gni100) reconnecting <6>[2058805.590268] Lustre: atlas1-MDT0000: Connection restored to 4e662dec-f2ca-3ff6-894a-e4154254c4ca (at 11353@gni100) <4>[2058823.381093] Lustre: atlas1-MDT0000: Client ac666f6b-6955-d675-ff5c-b6635e686fb5 (at 11354@gni100) reconnecting <6>[2058823.392939] Lustre: atlas1-MDT0000: Connection restored to ac666f6b-6955-d675-ff5c-b6635e686fb5 (at 11354@gni100) <4>[2058889.364077] Lustre: atlas1-MDT0000: Client 74160213-e786-c158-1ced-c8d4a870bb93 (at 11397@gni100) reconnecting <6>[2058889.375926] Lustre: atlas1-MDT0000: Connection restored to 74160213-e786-c158-1ced-c8d4a870bb93 (at 11397@gni100) <6>[2058889.388062] Lustre: Skipped 1 previous similar message <4>[2059039.276904] Lustre: atlas1-MDT0000: Client f744911b-501b-3230-cb3b-32431c04596c (at 11408@gni100) reconnecting <4>[2059039.288742] Lustre: Skipped 10 previous similar messages <6>[2059039.295121] Lustre: atlas1-MDT0000: Connection restored to f744911b-501b-3230-cb3b-32431c04596c (at 11408@gni100) <6>[2059039.307231] Lustre: Skipped 9 previous similar messages <4>[2059091.335853] Lustre: atlas1-MDT0000: Client 89256a0d-c0b7-be9f-503d-3d0a81d1c177 (at 11359@gni100) reconnecting <4>[2059091.347684] Lustre: Skipped 1 previous similar message <6>[2059091.353887] Lustre: atlas1-MDT0000: Connection restored to 89256a0d-c0b7-be9f-503d-3d0a81d1c177 (at 11359@gni100) <6>[2059091.366029] Lustre: Skipped 1 previous similar message <4>[2059123.374343] Lustre: atlas1-MDT0000: Client ac666f6b-6955-d675-ff5c-b6635e686fb5 (at 11354@gni100) reconnecting <6>[2059123.386214] Lustre: atlas1-MDT0000: Connection restored to ac666f6b-6955-d675-ff5c-b6635e686fb5 (at 11354@gni100) <4>[2059186.901251] Lustre: atlas1-MDT0000: Client 34c7726b-7e28-ce42-8a47-ad214e65a4f8 (at 12883@gni100) reconnecting <6>[2059186.913117] Lustre: atlas1-MDT0000: Connection restored to 34c7726b-7e28-ce42-8a47-ad214e65a4f8 (at 12883@gni100) <4>[2059271.270255] Lustre: atlas1-MDT0000: Client f9c05f17-90b9-a21c-43c1-c1abebbcff02 (at 17549@gni100) reconnecting <4>[2059271.282171] Lustre: Skipped 1 previous similar message <6>[2059271.288358] Lustre: atlas1-MDT0000: Connection restored to f9c05f17-90b9-a21c-43c1-c1abebbcff02 (at 17549@gni100) <6>[2059271.300503] Lustre: Skipped 1 previous similar message <4>[2059403.944191] Lustre: atlas1-MDT0000: Client 45136629-82f6-fe07-9605-a8d9620762e2 (at 11634@gni100) reconnecting <4>[2059403.956023] Lustre: Skipped 9 previous similar messages <6>[2059403.962296] Lustre: atlas1-MDT0000: Connection restored to 45136629-82f6-fe07-9605-a8d9620762e2 (at 11634@gni100) <6>[2059403.974419] Lustre: Skipped 9 previous similar messages <4>[2059654.580080] Lustre: 14923:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1488553900/real 1488553900] req@ffff881a4cb5e680 x1558716322542112/t0(0) o6->atlas1-OST03ad-osc-MDT0000@10.36.225.107@o2ib:28/4 lens 664/432 e 12 to 1 dl 1488554750 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[2059654.613424] Lustre: atlas1-OST03ad-osc-MDT0000: Connection to atlas1-OST03ad (at 10.36.225.107@o2ib) was lost; in progress operations using this service will wait for recovery to complete <4>[2059659.624048] Lustre: 14925:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1488553900/real 1488553900] req@ffff88236c1466c0 x1558716322542068/t0(0) o6->atlas1-OST0214-osc-MDT0000@10.36.225.130@o2ib:28/4 lens 664/432 e 12 to 1 dl 1488554750 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[2059659.657313] Lustre: atlas1-OST0214-osc-MDT0000: Connection to atlas1-OST0214 (at 10.36.225.130@o2ib) was lost; in progress operations using this service will wait for recovery to complete <4>[2059679.845403] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2059714.566999] Lustre: 14922:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1488553900/real 1488553900] req@ffff881f2d5e6680 x1558716322542044/t0(0) o6->atlas1-OST00cc-osc-MDT0000@10.36.225.90@o2ib:28/4 lens 664/432 e 12 to 1 dl 1488554750 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[2059714.600178] Lustre: atlas1-OST00cc-osc-MDT0000: Connection to atlas1-OST00cc (at 10.36.225.90@o2ib) was lost; in progress operations using this service will wait for recovery to complete <6>[2059714.722962] Lustre: atlas1-OST00e5-osc-MDT0000: Connection restored to 10.36.225.115@o2ib (at 10.36.225.115@o2ib) <6>[2059714.735088] Lustre: Skipped 15 previous similar messages <4>[2059901.658386] Lustre: atlas1-MDT0000: haven't heard from client 630be253-9b89-b1ca-3f18-a61b5bcc4419 (at 11405@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f4130e800, cur 1488555005 expire 1488554105 last 1488553653 <6>[2062696.546334] Lustre: atlas1-MDT0000: Connection restored to 6fb67104-2f37-eea9-7e8c-6d652191a710 (at 2024@gni100) <6>[2062696.558372] Lustre: Skipped 1 previous similar message <6>[2063793.819691] Lustre: atlas1-MDT0000: Connection restored to fe4106a2-718e-6704-39bf-927e31eede73 (at 6888@gni100) <6>[2063793.831748] Lustre: Skipped 43 previous similar messages <4>[2065641.289752] Lustre: 15639:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1488560177/real 1488560177] req@ffff88084ba82c80 x1558716331526504/t0(0) o104->atlas1-MDT0000@8111@gni100:15/16 lens 296/224 e 0 to 1 dl 1488560744 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[2065641.321181] Lustre: 15639:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 1 previous similar message <4>[2065669.248697] Lustre: 15859:0:(service.c:1336:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[2065669.248698] req@ffff882f47d76380 x1558697590250876/t0(0) o36->dd20153a-8521-3665-c176-6fbe4f953815@9228@gni100:512/0 lens 616/3128 e 12 to 0 dl 1488560777 ref 2 fl Interpret:/0/0 rc 0/0 <4>[2065924.286228] Lustre: atlas1-MDT0000: Client dd20153a-8521-3665-c176-6fbe4f953815 (at 9228@gni100) reconnecting <4>[2065924.297951] Lustre: Skipped 13 previous similar messages <6>[2065924.304320] Lustre: atlas1-MDT0000: Connection restored to dd20153a-8521-3665-c176-6fbe4f953815 (at 9228@gni100) <6>[2065924.316422] Lustre: Skipped 9 previous similar messages <4>[2066017.620872] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2066020.861223] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2066023.687692] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2066199.644779] Lustre: atlas1-MDT0000: haven't heard from client cb844cf0-d8f1-2632-186a-cf0b5d4e0ce3 (at 8111@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f12976400, cur 1488561303 expire 1488560403 last 1488559951 <4>[2066199.670050] Lustre: Skipped 4 previous similar messages <3>[2066199.676349] LustreError: 15639:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) ### client (nid 8111@gni100) failed to reply to blocking AST (req status 0 rc -5), evict it ns: mdt-atlas1-MDT0000_UUID lock: ffff883edbfe9b40/0xd94b677bb428e7ba lrc: 4/0,0 mode: PR/PR res: [0x200259393:0x2574:0x0].0x0 bits 0x13 rrc: 3 type: IBT flags: 0x60200400000020 nid: 8111@gni100 remote: 0x620022166c403f8a expref: 14 pid: 16180 timeout: 6361210292 lvb_type: 0 <3>[2066199.721431] LustreError: 15639:0:(ldlm_lockd.c:689:ldlm_handle_ast_error()) Skipped 1 previous similar message <3>[2066199.733270] LustreError: 138-a: atlas1-MDT0000: A client on nid 8111@gni100 was evicted due to a lock blocking callback time out: rc -5 <3>[2066199.747521] LustreError: Skipped 1 previous similar message <4>[2066199.755511] Lustre: 15639:0:(service.c:2097:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:526s); client may timeout. req@ffff882f47d76380 x1558697590250876/t652737897452(0) o36->dd20153a-8521-3665-c176-6fbe4f953815@9228@gni100:512/0 lens 616/424 e 12 to 0 dl 1488560777 ref 1 fl Complete:/0/0 rc 0/0 <3>[2067164.698540] LustreError: 15957:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a3f4d:0x17fb6:0x0]: rc = -2 <4>[2071091.830755] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2075360.628886] Lustre: atlas1-MDT0000: haven't heard from client a6b1d234-ce68-14f6-e917-e28e43e22b0b (at 5490@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0d64c000, cur 1488570464 expire 1488569564 last 1488569112 <4>[2080418.114740] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2080421.242170] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2080423.374306] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2082820.815577] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2094549.799946] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2099946.594176] Lustre: atlas1-MDT0000: haven't heard from client b64f1645-c536-5d41-1cab-3c70b288580f (at 8335@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883ecfa9b000, cur 1488595050 expire 1488594150 last 1488593698 <4>[2100650.306278] Lustre: atlas1-MDT0000: Client 0c782632-bf75-f869-2387-5c4153b90830 (at 10.38.144.46@o2ib5) reconnecting <6>[2100650.318773] Lustre: atlas1-MDT0000: Connection restored to 0c782632-bf75-f869-2387-5c4153b90830 (at 10.38.144.46@o2ib5) <4>[2101217.355486] Lustre: atlas1-MDT0000: Client 0c782632-bf75-f869-2387-5c4153b90830 (at 10.38.144.46@o2ib5) reconnecting <6>[2101217.367941] Lustre: atlas1-MDT0000: Connection restored to 0c782632-bf75-f869-2387-5c4153b90830 (at 10.38.144.46@o2ib5) <4>[2105961.786478] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2113515.586316] Lustre: atlas1-MDT0000: haven't heard from client 9876f080-be36-de78-ead6-e66a7816d4ee (at 11580@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f1111c400, cur 1488608619 expire 1488607719 last 1488607267 <4>[2117613.693156] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2128785.758930] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2138025.283831] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2138028.694955] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2138030.751511] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2140197.741969] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2150054.529958] Lustre: atlas1-MDT0000: haven't heard from client 55d78bfe-cbc9-5046-61b3-4d1cee8a47ca (at 15318@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f11c8d800, cur 1488645158 expire 1488644258 last 1488643806 <4>[2151182.525909] Lustre: atlas1-MDT0000: haven't heard from client 10074592-5a89-c8f0-74b7-bc8a21a4ded1 (at 4032@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883ed25fd800, cur 1488646286 expire 1488645386 last 1488644934 <4>[2151926.727321] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2152418.167294] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2152421.139128] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2152423.168852] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2154109.524828] Lustre: atlas1-MDT0000: haven't heard from client 7299748b-885f-4033-adec-11f1de3e0b54 (at 11607@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0fe9b800, cur 1488649213 expire 1488648313 last 1488647861 <4>[2158603.514586] Lustre: atlas1-MDT0000: haven't heard from client 2553266c-09a3-c717-ac93-999cb494fc9d (at 234@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880316a1c800, cur 1488653707 expire 1488652807 last 1488652355 <4>[2159054.514612] Lustre: atlas1-MDT0000: haven't heard from client fae85206-a19f-f8f8-d1a0-7a9e20d0ae38 (at 5167@gni100) in 1204 seconds. I think it's dead, and I am evicting it. exp ffff883f0b7e0000, cur 1488654158 expire 1488653258 last 1488652954 <4>[2163338.711394] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2166819.107382] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2166822.266520] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2166824.415474] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <3>[2169507.896461] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 14416@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff8804412cfbc0/0xd94b6782abc9f6cd lrc: 4/0,0 mode: PR/PR res: [0x200395934:0x12c47:0x0].0x0 bits 0x1b rrc: 3 type: IBT flags: 0x60200400000020 nid: 14416@gni100 remote: 0xbb619583397a4daa expref: 22 pid: 16343 timeout: 6464176812 lvb_type: 0 <3>[2169507.939944] LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) Skipped 1 previous similar message <4>[2173144.122664] Lustre: 14925:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1488667397/real 1488667397] req@ffff880fb9929cc0 x1558716472735772/t0(0) o6->atlas1-OST03c9-osc-MDT0000@10.36.225.135@o2ib:28/4 lens 664/432 e 12 to 1 dl 1488668247 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[2173144.155934] Lustre: atlas1-OST03c9-osc-MDT0000: Connection to atlas1-OST03c9 (at 10.36.225.135@o2ib) was lost; in progress operations using this service will wait for recovery to complete <4>[2173144.175443] Lustre: Skipped 1 previous similar message <6>[2173144.275204] Lustre: atlas1-OST03c9-osc-MDT0000: Connection restored to 10.36.225.135@o2ib (at 10.36.225.135@o2ib) <4>[2173144.666656] Lustre: 14923:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1488667397/real 1488667397] req@ffff883ee6db30c0 x1558716472735768/t0(0) o6->atlas1-OST01a2-osc-MDT0000@10.36.225.160@o2ib:28/4 lens 664/432 e 12 to 1 dl 1488668247 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[2173144.699920] Lustre: atlas1-OST01a2-osc-MDT0000: Connection to atlas1-OST01a2 (at 10.36.225.160@o2ib) was lost; in progress operations using this service will wait for recovery to complete <6>[2173144.813327] Lustre: atlas1-OST00c8-osc-MDT0000: Connection restored to 10.36.225.86@o2ib (at 10.36.225.86@o2ib) <4>[2173541.501514] Lustre: atlas1-MDT0000: haven't heard from client a7ae752c-c109-7d99-62aa-5c7b23dec77c (at 4184@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f11e86c00, cur 1488668645 expire 1488667745 last 1488667293 <4>[2175067.695484] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2179088.488817] Lustre: atlas1-MDT0000: haven't heard from client b267f9eb-fe11-e49f-0053-32c7f36f03ec (at 7361@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f1060e800, cur 1488674192 expire 1488673292 last 1488672840 <4>[2186479.679831] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2189066.694018] Lustre: atlas1-MDT0000: Client b3c5618a-c333-0113-1a4b-1c5951f265d5 (at 10.36.205.218@o2ib) reconnecting <6>[2189066.706445] Lustre: atlas1-MDT0000: Connection restored to b3c5618a-c333-0113-1a4b-1c5951f265d5 (at 10.36.205.218@o2ib) <6>[2189066.719137] Lustre: Skipped 1 previous similar message <4>[2189650.205249] Lustre: atlas1-MDT0000: Client b3c5618a-c333-0113-1a4b-1c5951f265d5 (at 10.36.205.218@o2ib) reconnecting <6>[2189650.217668] Lustre: atlas1-MDT0000: Connection restored to b3c5618a-c333-0113-1a4b-1c5951f265d5 (at 10.36.205.218@o2ib) <4>[2197891.664250] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2199690.460511] Lustre: atlas1-MDT0000: haven't heard from client d564eb3b-2eb4-5f85-a382-308e7704ca91 (at 6707@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883ea10da000, cur 1488694794 expire 1488693894 last 1488693442 <4>[2203858.205681] Lustre: atlas1-MDT0000: Client cba12965-15ac-97b8-c690-209d627234d1 (at 2248@gni100) reconnecting <6>[2203858.217419] Lustre: atlas1-MDT0000: Connection restored to cba12965-15ac-97b8-c690-209d627234d1 (at 2248@gni100) <4>[2209303.649819] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2220715.635407] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <6>[2221899.375740] Lustre: atlas1-MDT0000: Connection restored to 10.36.226.117@o2ib (at 0@lo) <6>[2221899.385468] Lustre: client wants to enable acl, but mdt not! <4>[2221899.575379] Lustre: Mounted atlas1-client <4>[2224425.225179] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2224428.607424] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2224431.040493] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <3>[2226399.233294] LustreError: 15578:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003a392b:0xfd81:0x0]: rc = -2 <3>[2226399.248315] LustreError: 15578:0:(mdt_handler.c:893:mdt_getattr_internal()) Skipped 2 previous similar messages <4>[2227620.411132] Lustre: Unmounted atlas1-client <3>[2229910.018630] LustreError: 16228:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003877bb:0xd677:0x0]: rc = -2 <3>[2229973.874403] LustreError: 16217:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003829dd:0x11c08:0x0]: rc = -2 <3>[2230065.758780] LustreError: 16134:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003820ae:0x48d7:0x0]: rc = -2 <3>[2230069.055437] LustreError: 16356:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003863b4:0xc7fd:0x0]: rc = -2 <3>[2230201.857682] LustreError: 16057:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x200373720:0x1d1:0x0]: rc = -2 <3>[2230268.058724] LustreError: 16428:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2003749e4:0x2e4:0x0]: rc = -2 <4>[2233078.618044] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2238819.695948] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2238822.664106] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2241382.402002] Lustre: atlas1-MDT0000: haven't heard from client 4338483a-24c6-0428-0082-89a639c5f877 (at 18115@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0fcad800, cur 1488736486 expire 1488735586 last 1488735134 <4>[2241382.427422] Lustre: Skipped 1 previous similar message <4>[2241833.409527] Lustre: atlas1-MDT0000: haven't heard from client c7858274-fced-6211-aaca-c0ff53381418 (at 8282@gni100) in 985 seconds. I think it's dead, and I am evicting it. exp ffff883ed1798800, cur 1488736937 expire 1488736037 last 1488735952 <4>[2244490.602464] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2253220.467259] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2253223.582542] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2253225.725831] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2255585.589124] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2266680.572325] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2273672.377408] Lustre: atlas1-MDT0000: haven't heard from client 40600914-d596-ef15-fa6b-8e9d4a88d71a (at 16285@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0f50ac00, cur 1488768776 expire 1488767876 last 1488767424 <4>[2277458.557770] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2277745.252253] Lustre: atlas1-MDT0000: Client 596e4cbb-f4f3-d575-949d-20111e7d97c8 (at 8391@gni100) reconnecting <6>[2277745.260571] Lustre: atlas1-MDT0000: Connection restored to efe10c02-9984-36a8-c61e-d891297d6a91 (at 4623@gni100) <4>[2277745.276001] Lustre: Skipped 1 previous similar message <4>[2277769.329380] Lustre: atlas1-MDT0000: Client 60634e7e-c273-cf21-1d7f-917fc5220702 (at 3967@gni100) reconnecting <6>[2277769.341140] Lustre: atlas1-MDT0000: Connection restored to 60634e7e-c273-cf21-1d7f-917fc5220702 (at 3967@gni100) <6>[2277769.353153] Lustre: Skipped 1 previous similar message <4>[2277792.639447] Lustre: atlas1-MDT0000: Client aa900c38-cc0c-43cf-0f21-556af882a681 (at 5206@gni100) reconnecting <4>[2277792.651187] Lustre: Skipped 6 previous similar messages <6>[2277792.657474] Lustre: atlas1-MDT0000: Connection restored to aa900c38-cc0c-43cf-0f21-556af882a681 (at 5206@gni100) <6>[2277792.669542] Lustre: Skipped 6 previous similar messages <4>[2277815.464932] Lustre: atlas1-MDT0000: Client 23071002-d267-6871-42b5-4c72fed697ad (at 2754@gni100) reconnecting <4>[2277815.476659] Lustre: Skipped 3 previous similar messages <6>[2277815.482929] Lustre: atlas1-MDT0000: Connection restored to 23071002-d267-6871-42b5-4c72fed697ad (at 2754@gni100) <6>[2277815.494984] Lustre: Skipped 2 previous similar messages <4>[2277832.440712] Lustre: atlas1-MDT0000: Client 3ef689ab-002b-49f9-d536-9b6f8bfa040f (at 9947@gni100) reconnecting <4>[2277832.452439] Lustre: Skipped 2 previous similar messages <6>[2277832.458701] Lustre: atlas1-MDT0000: Connection restored to 3ef689ab-002b-49f9-d536-9b6f8bfa040f (at 9947@gni100) <6>[2277832.470711] Lustre: Skipped 2 previous similar messages <4>[2277861.815161] Lustre: atlas1-MDT0000: Client 1ddff9b2-f2ff-8cf1-54c2-bd3d00fc17b6 (at 8387@gni100) reconnecting <4>[2277861.826883] Lustre: Skipped 6 previous similar messages <6>[2277861.833152] Lustre: atlas1-MDT0000: Connection restored to 1ddff9b2-f2ff-8cf1-54c2-bd3d00fc17b6 (at 8387@gni100) <6>[2277861.845165] Lustre: Skipped 6 previous similar messages <4>[2277889.426606] Lustre: atlas1-MDT0000: Client b0e026fd-0d6c-7240-e33c-5815a928fc9c (at 5213@gni100) reconnecting <4>[2277889.438337] Lustre: Skipped 2 previous similar messages <6>[2277889.444606] Lustre: atlas1-MDT0000: Connection restored to b0e026fd-0d6c-7240-e33c-5815a928fc9c (at 5213@gni100) <6>[2277889.456616] Lustre: Skipped 2 previous similar messages <4>[2277948.925602] Lustre: atlas1-MDT0000: Client 61519b2c-203a-d1a2-51e8-981b5d8b5bac (at 2731@gni100) reconnecting <4>[2277948.937331] Lustre: Skipped 5 previous similar messages <6>[2277948.943624] Lustre: atlas1-MDT0000: Connection restored to 61519b2c-203a-d1a2-51e8-981b5d8b5bac (at 2731@gni100) <6>[2277948.955656] Lustre: Skipped 6 previous similar messages <4>[2278047.468901] Lustre: atlas1-MDT0000: Client a5cbd4e6-3e3a-6737-ed50-e001fea24674 (at 8384@gni100) reconnecting <4>[2278047.480634] Lustre: Skipped 10 previous similar messages <6>[2278047.487000] Lustre: atlas1-MDT0000: Connection restored to a5cbd4e6-3e3a-6737-ed50-e001fea24674 (at 8384@gni100) <6>[2278047.499015] Lustre: Skipped 9 previous similar messages <4>[2278191.315033] Lustre: atlas1-MDT0000: Client 7e65f1db-8237-3bb5-0468-28ea9bf4408a (at 4637@gni100) reconnecting <4>[2278191.326805] Lustre: Skipped 26 previous similar messages <6>[2278191.333203] Lustre: atlas1-MDT0000: Connection restored to 7e65f1db-8237-3bb5-0468-28ea9bf4408a (at 4637@gni100) <6>[2278191.345230] Lustre: Skipped 26 previous similar messages <4>[2278467.446670] Lustre: atlas1-MDT0000: Client e222ff5d-e0e0-ac99-7242-78558eeb5b7c (at 6078@gni100) reconnecting <4>[2278467.458397] Lustre: Skipped 28 previous similar messages <6>[2278467.464801] Lustre: atlas1-MDT0000: Connection restored to 42256dcd-fb4e-fd04-c6fd-37f5f6f763b9 (at 6078@gni100) <6>[2278467.476820] Lustre: Skipped 28 previous similar messages <4>[2278729.616495] Lustre: 14923:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1488772983/real 1488772983] req@ffff8827eccab6c0 x1558716822499868/t0(0) o6->atlas1-OST0398-osc-MDT0000@10.36.225.86@o2ib:28/4 lens 664/432 e 12 to 1 dl 1488773833 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[2278729.649689] Lustre: 14923:0:(client.c:2063:ptlrpc_expire_one_request()) Skipped 1 previous similar message <4>[2278729.661171] Lustre: atlas1-OST0398-osc-MDT0000: Connection to atlas1-OST0398 (at 10.36.225.86@o2ib) was lost; in progress operations using this service will wait for recovery to complete <4>[2278729.680666] Lustre: Skipped 1 previous similar message <4>[2278744.353330] Lustre: atlas1-MDT0000: haven't heard from client a4cc585c-3087-d86f-5f78-03247c93452c (at 6952@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f0d4d7400, cur 1488773848 expire 1488772948 last 1488772496 <4>[2278744.378672] Lustre: Skipped 2 previous similar messages <4>[2278980.003232] Lustre: atlas1-MDT0000: Client 8127528a-b038-ccd1-2e21-0e535b7b3f61 (at 1170@gni100) reconnecting <4>[2278980.014970] Lustre: Skipped 66 previous similar messages <6>[2278980.021413] Lustre: atlas1-MDT0000: Connection restored to 8127528a-b038-ccd1-2e21-0e535b7b3f61 (at 1170@gni100) <6>[2278980.033433] Lustre: Skipped 75 previous similar messages <4>[2279195.353670] Lustre: atlas1-MDT0000: haven't heard from client 74b88bb8-afd6-d19b-d047-1f55341f3578 (at 5203@gni100) in 1076 seconds. I think it's dead, and I am evicting it. exp ffff883f11383800, cur 1488774299 expire 1488773399 last 1488773223 <4>[2279195.378952] Lustre: Skipped 17 previous similar messages <6>[2279590.425975] Lustre: atlas1-MDT0000: Connection restored to 374790f2-b002-6974-b3de-95882209f0b9 (at 1215@gni100) <6>[2279590.438070] Lustre: Skipped 10 previous similar messages <4>[2288870.541918] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2300023.491017] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2305253.318800] Lustre: atlas1-MDT0000: haven't heard from client a0fc5bab-4e9c-42ac-c2f5-ae7206bca116 (at 18414@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f12f8bc00, cur 1488800357 expire 1488799457 last 1488799005 <4>[2305253.344212] Lustre: Skipped 1 previous similar message <6>[2305827.280983] Lustre: atlas1-MDT0000: Connection restored to ff6f2388-c772-3ab3-66bf-522583635125 (at 10.39.232.66@o2ib6) <6>[2305827.293695] Lustre: Skipped 4 previous similar messages <6>[2307249.047616] Lustre: atlas1-MDT0000: Connection restored to 25dcd053-7c91-8816-4767-43af28c8aae7 (at 10.39.232.74@o2ib6) <6>[2307273.161460] Lustre: atlas1-MDT0000: Connection restored to 3d57a70f-1c2f-76c0-fd76-519b15090f1c (at 10.39.232.87@o2ib6) <6>[2307273.174313] Lustre: Skipped 3 previous similar messages <6>[2307401.243045] Lustre: atlas1-MDT0000: Connection restored to 28c9f276-54db-9571-2d2b-a259b0d9fed4 (at 10.39.232.67@o2ib6) <6>[2307401.255759] Lustre: Skipped 1 previous similar message <6>[2307897.442863] Lustre: atlas1-MDT0000: Connection restored to 9a212798-f1f4-8060-3aa8-dac397577d0c (at 10.39.232.63@o2ib6) <4>[2308221.314159] Lustre: atlas1-MDT0000: haven't heard from client 7cf2bc67-297b-359e-cc6c-8df4aa6a0a5f (at 10.39.232.88@o2ib6) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88324fa40c00, cur 1488803325 expire 1488802425 last 1488801973 <4>[2308983.319128] Lustre: atlas1-MDT0000: haven't heard from client a8225fd6-9f8e-a8e9-f3de-a1f5a961777f (at 10.39.232.63@o2ib6) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8818f0057400, cur 1488804087 expire 1488803187 last 1488802735 <4>[2308983.345198] Lustre: Skipped 6 previous similar messages <4>[2310824.789741] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2310828.159013] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2310830.225612] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2311694.510760] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2323106.495116] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2325221.069675] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2325224.143658] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <4>[2325226.261800] ACPI: No handler for Region [POWR] (ffff884027a692b8) [IPMI] <3>[2332180.807790] LustreError: 15601:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x200380927:0x8f68:0x0]: rc = -2 <3>[2332180.822817] LustreError: 15601:0:(mdt_handler.c:893:mdt_getattr_internal()) Skipped 1 previous similar message <3>[2332198.181877] LustreError: 15747:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x200380878:0x10b00:0x0]: rc = -2 <3>[2332216.868388] LustreError: 15923:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20038086e:0x12d97:0x0]: rc = -2 <3>[2332216.883621] LustreError: 15923:0:(mdt_handler.c:893:mdt_getattr_internal()) Skipped 1 previous similar message <3>[2332295.127742] LustreError: 15532:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x200380878:0x9712:0x0]: rc = -2 <3>[2332295.142868] LustreError: 15532:0:(mdt_handler.c:893:mdt_getattr_internal()) Skipped 4 previous similar messages <3>[2332323.480162] LustreError: 15747:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x200380875:0x80d0:0x0]: rc = -2 <3>[2332323.495195] LustreError: 15747:0:(mdt_handler.c:893:mdt_getattr_internal()) Skipped 3 previous similar messages <3>[2332404.689857] LustreError: 15728:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x20038086e:0x1a110:0x0]: rc = -2 <3>[2332404.704986] LustreError: 15728:0:(mdt_handler.c:893:mdt_getattr_internal()) Skipped 1 previous similar message <4>[2334518.479345] Lustre: atlas1-MDT0000-o: trigger OI scrub by RPC for the [0x20003f37b:0x3e5:0x0] with flags 0x4a, rc = 0 <4>[2335639.271564] Lustre: atlas1-MDT0000: haven't heard from client f620af07-4f35-2483-a2e4-619f36c89f3a (at 10.39.232.93@o2ib6) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f17249000, cur 1488830743 expire 1488829843 last 1488829391 <3>[2335864.767537] LustreError: 16096:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2002ca8e6:0x82b4:0x0]: rc = -2 <4>[2336422.171009] Lustre: atlas1-MDT0000: Client 5ccc06f7-85c1-18f2-7f3a-55e4fc6242e9 (at 31@gni2) reconnecting <4>[2336422.182348] Lustre: Skipped 4 previous similar messages <6>[2336422.188629] Lustre: atlas1-MDT0000: Connection restored to 5ccc06f7-85c1-18f2-7f3a-55e4fc6242e9 (at 31@gni2) <4>[2337037.270593] Lustre: atlas1-MDT0000: haven't heard from client e5aa8ed7-adb1-3482-3911-c8fbf7bc1b46 (at 10.39.232.96@o2ib6) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88111d070800, cur 1488832141 expire 1488831241 last 1488830789 <3>[2338394.036116] LustreError: 16379:0:(mdt_handler.c:893:mdt_getattr_internal()) atlas1-MDT0000: getattr error for [0x2002caab8:0xf425:0x0]: rc = -2 <0>[2339122.255892] LustreError: 16352:0:(osd_handler.c:1610:osd_object_release()) LBUG <4>[2339122.264739] Pid: 16352, comm: mdt01_381 <4>[2339122.269446] <4>[2339122.269446] Call Trace: <4>[2339122.274687] [] libcfs_debug_dumpstack+0x55/0x80 [libcfs] <4>[2339122.282893] [] lbug_with_loc+0x47/0xb0 [libcfs] <4>[2339122.290225] [] osd_object_release+0x88/0x90 [osd_ldiskfs] <4>[2339122.298765] [] lu_object_put+0x16d/0x3b0 [obdclass] <4>[2339122.306500] [] mdt_getattr_name_lock+0x5f7/0x1900 [mdt] <4>[2339122.314609] [] mdt_intent_getattr+0x292/0x470 [mdt] <4>[2339122.322330] [] mdt_intent_policy+0x4be/0xc70 [mdt] <4>[2339122.329981] [] ldlm_lock_enqueue+0x127/0x990 [ptlrpc] <4>[2339122.337912] [] ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] <4>[2339122.346565] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] <4>[2339122.354501] [] tgt_enqueue+0x61/0x230 [ptlrpc] <4>[2339122.361753] [] tgt_request_handle+0x8ec/0x1440 [ptlrpc] <4>[2339122.369877] [] ptlrpc_main+0xd21/0x1800 [ptlrpc] <4>[2339122.377324] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] <4>[2339122.384747] [] kthread+0x9e/0xc0 <4>[2339122.390625] [] child_rip+0xa/0x20 <4>[2339122.396587] [] ? kthread+0x0/0xc0 <4>[2339122.402556] [] ? child_rip+0x0/0x20 <4>[2339122.408719] <0>[2339122.421523] Kernel panic - not syncing: LBUG <4>[2339122.426713] Pid: 16352, comm: mdt01_381 Not tainted 2.6.32-642.6.2.el6.atlas.x86_64 #1 <4>[2339122.436200] Call Trace: <4>[2339122.439356] [] ? panic+0xa7/0x179 <4>[2339122.445330] [] ? lbug_with_loc+0x9b/0xb0 [libcfs] <4>[2339122.452858] [] ? osd_object_release+0x88/0x90 [osd_ldiskfs] <4>[2339122.461590] [] ? lu_object_put+0x16d/0x3b0 [obdclass] <4>[2339122.469496] [] ? mdt_getattr_name_lock+0x5f7/0x1900 [mdt] <4>[2339122.478024] [] ? mdt_intent_getattr+0x292/0x470 [mdt] <4>[2339122.485941] [] ? mdt_intent_policy+0x4be/0xc70 [mdt] <4>[2339122.493783] [] ? ldlm_lock_enqueue+0x127/0x990 [ptlrpc] <4>[2339122.501911] [] ? ldlm_handle_enqueue0+0x807/0x14d0 [ptlrpc] <4>[2339122.510649] [] ? tgt_lookup_reply+0x31/0x190 [ptlrpc] <4>[2339122.518582] [] ? tgt_enqueue+0x61/0x230 [ptlrpc] <4>[2339122.526026] [] ? tgt_request_handle+0x8ec/0x1440 [ptlrpc] <4>[2339122.534559] [] ? ptlrpc_main+0xd21/0x1800 [ptlrpc] <4>[2339122.542187] [] ? ptlrpc_main+0x0/0x1800 [ptlrpc] <4>[2339122.549616] [] ? kthread+0x9e/0xc0 <4>[2339122.555681] [] ? child_rip+0xa/0x20 <4>[2339122.561845] [] ? kthread+0x0/0xc0 <4>[2339122.567837] [] ? child_rip+0x0/0x20