-- Logs begin at Mon 2020-10-19 09:51:41 PDT, end at Tue 2020-10-20 06:32:02 PDT. --
Oct 19 09:53:31 oak-io5-s2 kernel: Lustre: oak-OST00f2: Connection restored to 56c1c1d4-2c21-4 (at 10.51.2.8@o2ib3)
Oct 19 09:53:31 oak-io5-s2 kernel: Lustre: Skipped 966 previous similar messages
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00c6: deleting orphan objects from 0x3480000402:4338755 to 0x3480000402:4338785
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00d1: deleting orphan objects from 0x3740000402:4726472 to 0x3740000402:4726497
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00ca: deleting orphan objects from 0x3580000401:4709549 to 0x3580000401:4709569
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00e5: deleting orphan objects from 0x3c40000401:4516092 to 0x3c40000401:4516129
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00ea: deleting orphan objects from 0x3d80000401:4705139 to 0x3d80000401:4705185
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00f6: deleting orphan objects from 0x4080000401:4981204 to 0x4080000401:4981249
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00e9: deleting orphan objects from 0x3d40000402:4807481 to 0x3d40000402:4807521
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00f9: deleting orphan objects from 0x4140000401:5044262 to 0x4140000401:5044289
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00fd: deleting orphan objects from 0x4240000402:5211338 to 0x4240000402:5211361
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00f4: deleting orphan objects from 0x4000000401:5162965 to 0x4000000401:5163009
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST0103: deleting orphan objects from 0x43c0000401:5107876 to 0x43c0000401:5107905
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST010b: deleting orphan objects from 0x45c0000401:4746909 to 0x45c0000401:4746945
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00f3: deleting orphan objects from 0x3fc0000401:5093537 to 0x3fc0000401:5093569
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST010e: deleting orphan objects from 0x4680000401:4777845 to 0x4680000401:4777889
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST0104: deleting orphan objects from 0x4400000401:5081510 to 0x4400000401:5081537
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00cb: deleting orphan objects from 0x35c0000402:4745265 to 0x35c0000402:4745281
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00f7: deleting orphan objects from 0x40c0000401:4959023 to 0x40c0000401:4959041
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00e2: deleting orphan objects from 0x3b80000401:4657887 to 0x3b80000401:4657921
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00d2: deleting orphan objects from 0x3780000402:4620915 to 0x3780000402:4620961
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00de: deleting orphan objects from 0x3a80000401:4640511 to 0x3a80000401:4640545
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00e0: deleting orphan objects from 0x3b00000401:4648192 to 0x3b00000401:4648225
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00eb: deleting orphan objects from 0x3dc0000402:4786671 to 0x3dc0000402:4786689
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00e7: deleting orphan objects from 0x3cc0000401:4777818 to 0x3cc0000401:4777857
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00e8: deleting orphan objects from 0x3d00000402:4770168 to 0x3d00000402:4770209
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00e4: deleting orphan objects from 0x3c00000401:4679807 to 0x3c00000401:4679841
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00ef: deleting orphan objects from 0x3ec0000402:4681275 to 0x3ec0000402:4681313
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00f5: deleting orphan objects from 0x4040000401:5114058 to 0x4040000401:5114081
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00f1: deleting orphan objects from 0x3f40000401:4820864 to 0x3f40000401:4820897
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00f2: deleting orphan objects from 0x3f80000401:5156200 to 0x3f80000401:5156225
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00fc: deleting orphan objects from 0x4200000401:5063281 to 0x4200000401:5063297
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST0106: deleting orphan objects from 0x4480000401:4868594 to 0x4480000401:4868609
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST0107: deleting orphan objects from 0x44c0000401:4405835 to 0x44c0000401:4405857
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00ff: deleting orphan objects from 0x42c0000402:5159496 to 0x42c0000402:5159521
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00fb: deleting orphan objects from 0x41c0000401:5171948 to 0x41c0000401:5171969
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00d4: deleting orphan objects from 0x3800000402:4791778 to 0x3800000402:4791809
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST0102: deleting orphan objects from 0x4380000402:5086471 to 0x4380000402:5086497
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00c1: deleting orphan objects from 0x3340000402:4062063 to 0x3340000402:4062081
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00c5: deleting orphan objects from 0x3440000402:4073279 to 0x3440000402:4073313
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00dc: deleting orphan objects from 0x3a00000402:4728450 to 0x3a00000402:4728481
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST0101: deleting orphan objects from 0x4340000402:4784861 to 0x4340000402:4784897
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00e6: deleting orphan objects from 0x3c80000401:4766000 to 0x3c80000401:4766017
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST010a: deleting orphan objects from 0x4580000401:4796469 to 0x4580000401:4796513
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST0105: deleting orphan objects from 0x4440000402:5120250 to 0x4440000402:5120289
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00c3: deleting orphan objects from 0x33c0000401:3970036 to 0x33c0000401:3970081
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00d6: deleting orphan objects from 0x3880000401:4598727 to 0x3880000401:4598753
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00df: deleting orphan objects from 0x3ac0000402:4785052 to 0x3ac0000402:4785089
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00c9: deleting orphan objects from 0x3540000402:4223284 to 0x3540000402:4223329
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00da: deleting orphan objects from 0x3980000402:4781946 to 0x3980000402:4781985
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00d7: deleting orphan objects from 0x38c0000401:4570940 to 0x38c0000401:4570977
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST0109: deleting orphan objects from 0x4540000402:4898846 to 0x4540000402:4898881
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST0100: deleting orphan objects from 0x4300000401:5088614 to 0x4300000401:5088641
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST010d: deleting orphan objects from 0x4640000402:4874366 to 0x4640000402:4874401
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00cf: deleting orphan objects from 0x36c0000402:4691521 to 0x36c0000402:4691553
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00fa: deleting orphan objects from 0x4180000402:5132113 to 0x4180000402:5132129
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00d9: deleting orphan objects from 0x3940000402:4737298 to 0x3940000402:4737313
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00f8: deleting orphan objects from 0x4100000402:5099746 to 0x4100000402:5099777
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST010f: deleting orphan objects from 0x46c0000401:4837090 to 0x46c0000401:4837121
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST010c: deleting orphan objects from 0x4600000402:4823307 to 0x4600000402:4823329
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00ee: deleting orphan objects from 0x3e80000402:4506892 to 0x3e80000402:4506913
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST0108: deleting orphan objects from 0x4500000401:4890156 to 0x4500000401:4890177
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00ce: deleting orphan objects from 0x3680000401:4243669 to 0x3680000401:4243745
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00d0: deleting orphan objects from 0x3700000401:4732727 to 0x3700000401:4732769
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00cc: deleting orphan objects from 0x3600000401:4688743 to 0x3600000401:4688769
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00d8: deleting orphan objects from 0x3900000402:4730349 to 0x3900000402:4730369
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00c0: deleting orphan objects from 0x3300000402:4137048 to 0x3300000402:4137089
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00dd: deleting orphan objects from 0x3a40000402:4679016 to 0x3a40000402:4679041
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00e1: deleting orphan objects from 0x3b40000401:4720050 to 0x3b40000401:4720065
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00c7: deleting orphan objects from 0x34c0000402:4243554 to 0x34c0000402:4243585
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00db: deleting orphan objects from 0x39c0000401:4501718 to 0x39c0000401:4501761
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00e3: deleting orphan objects from 0x3bc0000402:4696243 to 0x3bc0000402:4696289
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00ec: deleting orphan objects from 0x3e00000401:4551959 to 0x3e00000401:4552001
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00c4: deleting orphan objects from 0x3400000402:4241699 to 0x3400000402:4241729
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00d3: deleting orphan objects from 0x37c0000401:4719963 to 0x37c0000401:4720001
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00c2: deleting orphan objects from 0x3380000402:3943757 to 0x3380000402:3943777
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00d5: deleting orphan objects from 0x3840000401:4605439 to 0x3840000401:4605473
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00f0: deleting orphan objects from 0x3f00000402:4683169 to 0x3f00000402:4683201
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00ed: deleting orphan objects from 0x3e40000402:4765691 to 0x3e40000402:4765729
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00cd: deleting orphan objects from 0x3640000402:4678565 to 0x3640000402:4678593
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00c8: deleting orphan objects from 0x3500000401:4187451 to 0x3500000401:4187489
Oct 19 09:53:40 oak-io5-s2 kernel: Lustre: oak-OST00fe: deleting orphan objects from 0x4280000401:5074790 to 0x4280000401:5074817
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00cd: deleting orphan objects from 0x3640000401:2585534 to 0x3640000401:2585569
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00c6: deleting orphan objects from 0x3480000401:2054302 to 0x3480000401:2054337
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00c7: deleting orphan objects from 0x34c0000401:1980521 to 0x34c0000401:1980609
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00eb: deleting orphan objects from 0x3dc0000401:2629469 to 0x3dc0000401:2629537
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00dc: deleting orphan objects from 0x3a00000401:2602294 to 0x3a00000401:2602433
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00ea: deleting orphan objects from 0x3d80000400:2602480 to 0x3d80000400:2602721
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00cb: deleting orphan objects from 0x35c0000401:2620348 to 0x35c0000401:2620385
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00df: deleting orphan objects from 0x3ac0000401:2639147 to 0x3ac0000401:2639201
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00e5: deleting orphan objects from 0x3c40000400:2456631 to 0x3c40000400:2456705
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00ce: deleting orphan objects from 0x3680000400:2188060 to 0x3680000400:2188385
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00e2: deleting orphan objects from 0x3b80000400:2560231 to 0x3b80000400:2560289
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00d2: deleting orphan objects from 0x3780000401:2491278 to 0x3780000401:2491457
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST0103: deleting orphan objects from 0x43c0000400:2813963 to 0x43c0000400:2814113
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00d7: deleting orphan objects from 0x38c0000400:2472941 to 0x38c0000400:2475105
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00f2: deleting orphan objects from 0x3f80000400:2826422 to 0x3f80000400:2826529
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00f5: deleting orphan objects from 0x4040000400:2797754 to 0x4040000400:2798113
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00ed: deleting orphan objects from 0x3e40000401:2609103 to 0x3e40000401:2610785
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST0105: deleting orphan objects from 0x4440000401:2817913 to 0x4440000401:2817985
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00e0: deleting orphan objects from 0x3b00000400:2541144 to 0x3b00000400:2541217
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00fb: deleting orphan objects from 0x41c0000400:2836656 to 0x41c0000400:2836673
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00f1: deleting orphan objects from 0x3f40000400:2650204 to 0x3f40000400:2650241
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00f0: deleting orphan objects from 0x3f00000401:2591419 to 0x3f00000401:2591521
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00cc: deleting orphan objects from 0x3600000400:2529749 to 0x3600000400:2530113
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00e8: deleting orphan objects from 0x3d00000401:2636740 to 0x3d00000401:2636865
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00ee: deleting orphan objects from 0x3e80000401:2480995 to 0x3e80000401:2481377
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00e4: deleting orphan objects from 0x3c00000400:2571474 to 0x3c00000400:2571617
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00c4: deleting orphan objects from 0x3400000401:1986493 to 0x3400000401:1986529
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00c9: deleting orphan objects from 0x3540000401:1971813 to 0x3540000401:1971841
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST010c: deleting orphan objects from 0x4600000401:2842213 to 0x4600000401:2842305
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST010d: deleting orphan objects from 0x4640000401:2906850 to 0x4640000401:2906913
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00d4: deleting orphan objects from 0x3800000401:2633296 to 0x3800000401:2633441
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00d5: deleting orphan objects from 0x3840000402:2483450 to 0x3840000402:2483489
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00d8: deleting orphan objects from 0x3900000401:2562315 to 0x3900000401:2562465
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00c3: deleting orphan objects from 0x33c0000400:1825924 to 0x33c0000400:1825985
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00e7: deleting orphan objects from 0x3cc0000400:2639169 to 0x3cc0000400:2639201
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00de: deleting orphan objects from 0x3a80000400:2521746 to 0x3a80000400:2521985
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST0102: deleting orphan objects from 0x4380000401:2780114 to 0x4380000401:2780289
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00f6: deleting orphan objects from 0x4080000400:2752233 to 0x4080000400:2752353
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00c8: deleting orphan objects from 0x3500000400:1970872 to 0x3500000400:1970913
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST0104: deleting orphan objects from 0x4400000400:2813282 to 0x4400000400:2813409
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00f7: deleting orphan objects from 0x40c0000400:2692343 to 0x40c0000400:2692449
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00f9: deleting orphan objects from 0x4140000400:2748166 to 0x4140000400:2748609
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00c1: deleting orphan objects from 0x3340000401:1911830 to 0x3340000401:1911873
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00d0: deleting orphan objects from 0x3700000400:2616176 to 0x3700000400:2616481
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00c2: deleting orphan objects from 0x3380000401:1837302 to 0x3380000401:1837345
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST0100: deleting orphan objects from 0x4300000400:2804678 to 0x4300000400:2805057
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00dd: deleting orphan objects from 0x3a40000401:2563723 to 0x3a40000401:2563841
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00d6: deleting orphan objects from 0x3880000400:2526556 to 0x3880000400:2527009
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST010e: deleting orphan objects from 0x4680000400:2838641 to 0x4680000400:2838881
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00da: deleting orphan objects from 0x3980000401:2644601 to 0x3980000401:2645281
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00f3: deleting orphan objects from 0x3fc0000400:2797463 to 0x3fc0000400:2797505
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00fa: deleting orphan objects from 0x4180000401:2805332 to 0x4180000401:2806657
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST0109: deleting orphan objects from 0x4540000401:2903539 to 0x4540000401:2903873
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00d3: deleting orphan objects from 0x37c0000400:2611533 to 0x37c0000400:2611617
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00db: deleting orphan objects from 0x39c0000400:2492786 to 0x39c0000400:2492897
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00ff: deleting orphan objects from 0x42c0000401:2851777 to 0x42c0000401:2851905
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00c0: deleting orphan objects from 0x3300000401:1914272 to 0x3300000401:1914401
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST0106: deleting orphan objects from 0x4480000400:2891346 to 0x4480000400:2891457
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00f8: deleting orphan objects from 0x4100000401:2761021 to 0x4100000401:2761217
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00ca: deleting orphan objects from 0x3580000400:2592987 to 0x3580000400:2593281
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST010b: deleting orphan objects from 0x45c0000400:2790643 to 0x45c0000400:2791233
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST0101: deleting orphan objects from 0x4340000401:2604088 to 0x4340000401:2604161
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00d1: deleting orphan objects from 0x3740000401:2605040 to 0x3740000401:2605089
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00c5: deleting orphan objects from 0x3440000401:1907431 to 0x3440000401:1907585
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00d9: deleting orphan objects from 0x3940000401:2598648 to 0x3940000401:2598753
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00fc: deleting orphan objects from 0x4200000400:2756093 to 0x4200000400:2756289
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00fd: deleting orphan objects from 0x4240000401:2881890 to 0x4240000401:2881953
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST0107: deleting orphan objects from 0x44c0000400:2588293 to 0x44c0000400:2588321
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST0108: deleting orphan objects from 0x4500000400:2914832 to 0x4500000400:2915169
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00e9: deleting orphan objects from 0x3d40000401:2658376 to 0x3d40000401:2658465
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST010f: deleting orphan objects from 0x46c0000400:2871760 to 0x46c0000400:2871841
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00e3: deleting orphan objects from 0x3bc0000401:2584711 to 0x3bc0000401:2584737
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST010a: deleting orphan objects from 0x4580000400:2836316 to 0x4580000400:2836417
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00cf: deleting orphan objects from 0x36c0000401:2584406 to 0x36c0000401:2584481
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00e6: deleting orphan objects from 0x3c80000400:2633942 to 0x3c80000400:2634177
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00ec: deleting orphan objects from 0x3e00000400:2519263 to 0x3e00000400:2519617
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00f4: deleting orphan objects from 0x4000000400:2819188 to 0x4000000400:2819361
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00fe: deleting orphan objects from 0x4280000400:2800078 to 0x4280000400:2800385
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00ef: deleting orphan objects from 0x3ec0000401:2572557 to 0x3ec0000401:2572641
Oct 19 09:53:44 oak-io5-s2 kernel: Lustre: oak-OST00e1: deleting orphan objects from 0x3b40000400:2601181 to 0x3b40000400:2601217
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00d0: deleting orphan objects from 0x0:17011571 to 0x0:17011649
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00e3: deleting orphan objects from 0x0:16933706 to 0x0:16933729
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00cc: deleting orphan objects from 0x0:16579595 to 0x0:16579649
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00ce: deleting orphan objects from 0x0:14754372 to 0x0:14754401
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00ee: deleting orphan objects from 0x0:16097247 to 0x0:16097281
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00e2: deleting orphan objects from 0x0:16743589 to 0x0:16743617
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00ea: deleting orphan objects from 0x0:16953695 to 0x0:16953729
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00dd: deleting orphan objects from 0x0:16865395 to 0x0:16865441
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST0100: deleting orphan objects from 0x0:17782025 to 0x0:17782081
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST0107: deleting orphan objects from 0x0:16474538 to 0x0:16474561
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST010a: deleting orphan objects from 0x0:17351159 to 0x0:17351201
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00ef: deleting orphan objects from 0x0:16795629 to 0x0:16795649
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00d3: deleting orphan objects from 0x0:16961274 to 0x0:16961313
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST010f: deleting orphan objects from 0x0:17622795 to 0x0:17622817
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00f4: deleting orphan objects from 0x0:17899329 to 0x0:17899361
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST010b: deleting orphan objects from 0x0:17190427 to 0x0:17190465
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00e6: deleting orphan objects from 0x0:17159881 to 0x0:17159969
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00df: deleting orphan objects from 0x0:17162089 to 0x0:17162113
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00e7: deleting orphan objects from 0x0:17249932 to 0x0:17249953
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00d8: deleting orphan objects from 0x0:17000014 to 0x0:17000065
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST0102: deleting orphan objects from 0x0:17648795 to 0x0:17648833
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00de: deleting orphan objects from 0x0:16455558 to 0x0:16455585
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00db: deleting orphan objects from 0x0:16214789 to 0x0:16214817
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00fd: deleting orphan objects from 0x0:18194823 to 0x0:18194849
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00e9: deleting orphan objects from 0x0:17317781 to 0x0:17317825
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00d9: deleting orphan objects from 0x0:16965688 to 0x0:16965729
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST0108: deleting orphan objects from 0x0:17845917 to 0x0:17845953
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00ed: deleting orphan objects from 0x0:17083394 to 0x0:17083425
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST010c: deleting orphan objects from 0x0:17477957 to 0x0:17478017
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00c9: deleting orphan objects from 0x0:15702269 to 0x0:15702305
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00eb: deleting orphan objects from 0x0:17122442 to 0x0:17122465
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00cb: deleting orphan objects from 0x0:17040722 to 0x0:17040737
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00d2: deleting orphan objects from 0x0:16365037 to 0x0:16365089
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00f1: deleting orphan objects from 0x0:17241015 to 0x0:17241057
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00d6: deleting orphan objects from 0x0:16660399 to 0x0:16660417
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00fe: deleting orphan objects from 0x0:17741748 to 0x0:17741793
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST0109: deleting orphan objects from 0x0:17796524 to 0x0:17796545
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST0101: deleting orphan objects from 0x0:16939548 to 0x0:16939585
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00fa: deleting orphan objects from 0x0:17792556 to 0x0:17792577
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00f5: deleting orphan objects from 0x0:17782708 to 0x0:17782753
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00cf: deleting orphan objects from 0x0:16780861 to 0x0:16780897
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST010e: deleting orphan objects from 0x0:17486763 to 0x0:17486785
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00f7: deleting orphan objects from 0x0:17212078 to 0x0:17212097
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00ff: deleting orphan objects from 0x0:18047790 to 0x0:18047809
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00f2: deleting orphan objects from 0x0:18016487 to 0x0:18016545
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00c2: deleting orphan objects from 0x0:14743457 to 0x0:14743489
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00c5: deleting orphan objects from 0x0:15228945 to 0x0:15228961
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00fb: deleting orphan objects from 0x0:17980422 to 0x0:17980449
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00fc: deleting orphan objects from 0x0:17465649 to 0x0:17465729
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00e0: deleting orphan objects from 0x0:16653629 to 0x0:16653697
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00c4: deleting orphan objects from 0x0:15773696 to 0x0:15773729
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00e8: deleting orphan objects from 0x0:17137495 to 0x0:17137537
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00dc: deleting orphan objects from 0x0:16938224 to 0x0:16938241
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00c6: deleting orphan objects from 0x0:16267536 to 0x0:16267553
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00da: deleting orphan objects from 0x0:17231896 to 0x0:17231937
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00c0: deleting orphan objects from 0x0:15388973 to 0x0:15388993
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00c1: deleting orphan objects from 0x0:15235586 to 0x0:15235617
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00cd: deleting orphan objects from 0x0:16847853 to 0x0:16847873
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00e5: deleting orphan objects from 0x0:16172915 to 0x0:16172961
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00d5: deleting orphan objects from 0x0:15648953 to 0x0:15648993
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST0106: deleting orphan objects from 0x0:17681024 to 0x0:17681057
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00d1: deleting orphan objects from 0x0:17013859 to 0x0:17013889
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST0103: deleting orphan objects from 0x0:17817931 to 0x0:17817953
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00f0: deleting orphan objects from 0x0:16867899 to 0x0:16867969
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00f8: deleting orphan objects from 0x0:17608537 to 0x0:17608577
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00d4: deleting orphan objects from 0x0:17167653 to 0x0:17167745
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00ec: deleting orphan objects from 0x0:16357593 to 0x0:16357633
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00e4: deleting orphan objects from 0x0:16781183 to 0x0:16781217
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00d7: deleting orphan objects from 0x0:16314670 to 0x0:16314689
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00c7: deleting orphan objects from 0x0:15743764 to 0x0:15743809
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00f9: deleting orphan objects from 0x0:17526070 to 0x0:17526113
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00e1: deleting orphan objects from 0x0:16982657 to 0x0:16982689
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00c3: deleting orphan objects from 0x0:14625983 to 0x0:14626017
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00ca: deleting orphan objects from 0x0:16889819 to 0x0:16889857
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00c8: deleting orphan objects from 0x0:15581082 to 0x0:15581121
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST0104: deleting orphan objects from 0x0:17806842 to 0x0:17806881
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00f6: deleting orphan objects from 0x0:17518402 to 0x0:17518433
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST010d: deleting orphan objects from 0x0:17785117 to 0x0:17785153
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST0105: deleting orphan objects from 0x0:17928489 to 0x0:17928513
Oct 19 09:54:03 oak-io5-s2 kernel: Lustre: oak-OST00f3: deleting orphan objects from 0x0:17681936 to 0x0:17681953
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00df: deleting orphan objects from 0x3ac0000400:18399976 to 0x3ac0000400:18400001
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00eb: deleting orphan objects from 0x3dc0000400:18243003 to 0x3dc0000400:18243041
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00f7: deleting orphan objects from 0x40c0000402:18830767 to 0x40c0000402:18830785
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00e4: deleting orphan objects from 0x3c00000402:17996761 to 0x3c00000402:17996801
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST010e: deleting orphan objects from 0x4680000402:18926178 to 0x4680000402:18926241
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00e8: deleting orphan objects from 0x3d00000400:18398654 to 0x3d00000400:18398721
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST0100: deleting orphan objects from 0x4300000402:19563743 to 0x4300000402:19563777
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00d0: deleting orphan objects from 0x3700000402:18274482 to 0x3700000402:18274529
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00fe: deleting orphan objects from 0x4280000402:19507032 to 0x4280000402:19507137
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00d4: deleting orphan objects from 0x3800000400:18235856 to 0x3800000400:18235873
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00f4: deleting orphan objects from 0x4000000402:19670597 to 0x4000000402:19670753
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST0106: deleting orphan objects from 0x4480000402:19238862 to 0x4480000402:19238913
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00fd: deleting orphan objects from 0x4240000400:20029406 to 0x4240000400:20029441
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00d6: deleting orphan objects from 0x3880000402:17823436 to 0x3880000402:17823521
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00e5: deleting orphan objects from 0x3c40000402:17221397 to 0x3c40000402:17221441
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00fc: deleting orphan objects from 0x4200000402:19251827 to 0x4200000402:19251905
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00dc: deleting orphan objects from 0x3a00000400:17924355 to 0x3a00000400:17924417
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00f1: deleting orphan objects from 0x3f40000402:18401326 to 0x3f40000402:18401345
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00f2: deleting orphan objects from 0x3f80000402:19828494 to 0x3f80000402:19828641
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00fb: deleting orphan objects from 0x41c0000402:19774341 to 0x41c0000402:19774369
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00fa: deleting orphan objects from 0x4180000400:19615416 to 0x4180000400:19615521
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00d8: deleting orphan objects from 0x3900000400:18092472 to 0x3900000400:18092545
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00e6: deleting orphan objects from 0x3c80000402:18367510 to 0x3c80000402:18367649
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00ed: deleting orphan objects from 0x3e40000400:18318534 to 0x3e40000400:18318561
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST010b: deleting orphan objects from 0x45c0000402:18626746 to 0x45c0000402:18626785
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00e1: deleting orphan objects from 0x3b40000402:18183221 to 0x3b40000402:18183265
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00ec: deleting orphan objects from 0x3e00000402:17486917 to 0x3e00000402:17486977
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00f3: deleting orphan objects from 0x3fc0000402:19539450 to 0x3fc0000402:19539489
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00ee: deleting orphan objects from 0x3e80000400:17430619 to 0x3e80000400:17430689
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00c3: deleting orphan objects from 0x33c0000402:17918450 to 0x33c0000402:17918465
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00c9: deleting orphan objects from 0x3540000400:18876009 to 0x3540000400:18876033
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00d7: deleting orphan objects from 0x38c0000402:17456241 to 0x38c0000402:17456257
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST0103: deleting orphan objects from 0x43c0000402:19652665 to 0x43c0000402:19652705
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST0108: deleting orphan objects from 0x4500000402:19391925 to 0x4500000402:19392001
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00cd: deleting orphan objects from 0x3640000400:18029454 to 0x3640000400:18029473
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST0109: deleting orphan objects from 0x4540000400:19269933 to 0x4540000400:19269953
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00f5: deleting orphan objects from 0x4040000402:19551548 to 0x4040000402:19551585
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST010d: deleting orphan objects from 0x4640000400:19297628 to 0x4640000400:19297665
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00c0: deleting orphan objects from 0x3300000400:18549725 to 0x3300000400:18549793
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00ce: deleting orphan objects from 0x3680000402:15252693 to 0x3680000402:15252737
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00c8: deleting orphan objects from 0x3500000402:18719401 to 0x3500000402:18719553
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00d5: deleting orphan objects from 0x3840000400:16219960 to 0x3840000400:16220001
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00f9: deleting orphan objects from 0x4140000402:19284952 to 0x4140000402:19284993
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00d9: deleting orphan objects from 0x3940000400:18223709 to 0x3940000400:18223745
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00cf: deleting orphan objects from 0x36c0000400:17942548 to 0x36c0000400:17942593
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST0105: deleting orphan objects from 0x4440000400:19675163 to 0x4440000400:19675201
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00e7: deleting orphan objects from 0x3cc0000402:18409534 to 0x3cc0000402:18409569
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST0107: deleting orphan objects from 0x44c0000402:16873616 to 0x44c0000402:16873633
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00d1: deleting orphan objects from 0x3740000400:18162555 to 0x3740000400:18162593
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST0104: deleting orphan objects from 0x4400000402:19632786 to 0x4400000402:19632833
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00ff: deleting orphan objects from 0x42c0000400:19881150 to 0x42c0000400:19881185
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00e9: deleting orphan objects from 0x3d40000400:18501112 to 0x3d40000400:18501153
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00de: deleting orphan objects from 0x3a80000402:17578903 to 0x3a80000402:17578945
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00c4: deleting orphan objects from 0x3400000400:18942803 to 0x3400000400:18942881
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00ea: deleting orphan objects from 0x3d80000402:18192246 to 0x3d80000402:18192289
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00f8: deleting orphan objects from 0x4100000400:19468318 to 0x4100000400:19468353
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00f0: deleting orphan objects from 0x3f00000400:18032949 to 0x3f00000400:18032993
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00dd: deleting orphan objects from 0x3a40000400:17874616 to 0x3a40000400:17874657
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00f6: deleting orphan objects from 0x4080000402:19109713 to 0x4080000402:19109761
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00c6: deleting orphan objects from 0x3480000400:19652004 to 0x3480000400:19652065
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00c2: deleting orphan objects from 0x3380000400:17836647 to 0x3380000400:17836737
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST0102: deleting orphan objects from 0x4380000400:19334618 to 0x4380000400:19334657
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00db: deleting orphan objects from 0x39c0000402:17369834 to 0x39c0000402:17369857
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00c7: deleting orphan objects from 0x34c0000400:18983277 to 0x34c0000400:18983297
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00e3: deleting orphan objects from 0x3bc0000400:18068896 to 0x3bc0000400:18068929
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00ef: deleting orphan objects from 0x3ec0000400:17855601 to 0x3ec0000400:17855617
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00e0: deleting orphan objects from 0x3b00000402:17581551 to 0x3b00000402:17581569
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST010f: deleting orphan objects from 0x46c0000402:19100714 to 0x46c0000402:19100737
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00cc: deleting orphan objects from 0x3600000402:17760548 to 0x3600000402:17760609
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00cb: deleting orphan objects from 0x35c0000400:18293310 to 0x35c0000400:18293345
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00d2: deleting orphan objects from 0x3780000400:17347828 to 0x3780000400:17347905
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00c5: deleting orphan objects from 0x3440000400:18416460 to 0x3440000400:18416481
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST010c: deleting orphan objects from 0x4600000400:18926835 to 0x4600000400:18926881
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00c1: deleting orphan objects from 0x3340000400:18411253 to 0x3340000400:18411297
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00e2: deleting orphan objects from 0x3b80000402:17929014 to 0x3b80000402:17929057
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST0101: deleting orphan objects from 0x4340000400:18052881 to 0x4340000400:18052897
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00ca: deleting orphan objects from 0x3580000402:18060431 to 0x3580000402:18060513
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST010a: deleting orphan objects from 0x4580000402:18804815 to 0x4580000402:18804897
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00da: deleting orphan objects from 0x3980000400:18420126 to 0x3980000400:18420193
Oct 19 09:54:04 oak-io5-s2 kernel: Lustre: oak-OST00d3: deleting orphan objects from 0x37c0000402:18161706 to 0x37c0000402:18161729
Oct 19 09:54:20 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds
Oct 19 09:54:20 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.2.217@o2ib5 (2): c: 0, oc: 0, rc: 8
Oct 19 09:54:23 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds
Oct 19 09:54:23 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.2.217@o2ib5 (3): c: 0, oc: 0, rc: 8
Oct 19 09:54:26 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds
Oct 19 09:54:26 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.2.216@o2ib5 (1): c: 0, oc: 3, rc: 8
Oct 19 09:54:29 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds
Oct 19 09:54:29 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Skipped 1 previous similar message
Oct 19 09:54:29 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.2.217@o2ib5 (3): c: 0, oc: 0, rc: 8
Oct 19 09:54:29 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Skipped 1 previous similar message
Oct 19 09:54:38 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds
Oct 19 09:54:38 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Skipped 1 previous similar message
Oct 19 09:54:38 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.2.216@o2ib5 (1): c: 0, oc: 1, rc: 8
Oct 19 09:54:38 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Skipped 1 previous similar message
Oct 19 09:54:38 oak-io5-s2 kernel: LNetError: 28368:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to 10.51.13.2@o2ib3 from 10.0.2.118@o2ib5
Oct 19 09:54:38 oak-io5-s2 kernel: LNetError: 28368:0:(lib-move.c:1999:lnet_handle_find_routed_path()) Skipped 34 previous similar messages
Oct 19 09:54:38 oak-io5-s2 kernel: LNetError: 28368:0:(lib-move.c:2955:lnet_resend_pending_msgs_locked()) Error sending PUT to 12345-10.51.13.2@o2ib3: -113
Oct 19 09:54:48 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds
Oct 19 09:54:48 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.2.217@o2ib5 (3): c: 0, oc: 0, rc: 8
Oct 19 09:54:53 oak-io5-s2 kernel: Lustre: oak-OST00df: haven't heard from client b6e6a634-78c9-4 (at 10.51.1.10@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ecdce318c00, cur 1603126493 expire 1603126343 last 1603126266
Oct 19 09:54:53 oak-io5-s2 kernel: Lustre: Skipped 663 previous similar messages
Oct 19 09:55:08 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds
Oct 19 09:55:08 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Skipped 1 previous similar message
Oct 19 09:55:08 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.2.216@o2ib5 (1): c: 0, oc: 0, rc: 7
Oct 19 09:55:08 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Skipped 1 previous similar message
Oct 19 09:55:11 oak-io5-s2 kernel: LNetError: 28597:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to 10.51.2.23@o2ib3 from <?>
Oct 19 09:55:11 oak-io5-s2 kernel: LNetError: 28597:0:(lib-move.c:1999:lnet_handle_find_routed_path()) Skipped 10 previous similar messages
Oct 19 09:55:14 oak-io5-s2 kernel: LNetError: 28597:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to 10.51.1.19@o2ib3 from <?>
Oct 19 09:55:14 oak-io5-s2 kernel: LNetError: 28597:0:(lib-move.c:1999:lnet_handle_find_routed_path()) Skipped 271 previous similar messages
Oct 19 09:55:15 oak-io5-s2 kernel: LustreError: 84344:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8ecd27fd0050 x1678198916635280/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:322/0 lens 488/448 e 0 to 0 dl 1603126552 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 09:55:15 oak-io5-s2 kernel: Lustre: oak-OST00c1: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 09:55:15 oak-io5-s2 kernel: LustreError: 84344:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 1 previous similar message
Oct 19 09:55:15 oak-io5-s2 kernel: LustreError: 67799:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff8e6cc004c050 x1678198916913264/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:323/0 lens 488/448 e 0 to 0 dl 1603126553 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 09:55:15 oak-io5-s2 kernel: LustreError: 67799:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
Oct 19 09:55:17 oak-io5-s2 kernel: Lustre: 416804:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1603126505/real 1603126505]  req@ffff8edaa418bf00 x1680988519436672/t0(0) o104->oak-OST00d6@10.50.15.7@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1603126516 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Oct 19 09:55:17 oak-io5-s2 kernel: Lustre: 416804:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Oct 19 09:55:18 oak-io5-s2 kernel: Lustre: 420256:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1603126511/real 1603126511]  req@ffff8ecdc1cead00 x1680988519438144/t0(0) o106->oak-OST00f9@10.49.25.16@o2ib1:15/16 lens 296/280 e 0 to 1 dl 1603126518 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Oct 19 09:55:18 oak-io5-s2 kernel: Lustre: 420256:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
Oct 19 09:55:21 oak-io5-s2 kernel: Lustre: 420261:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1603126514/real 1603126514]  req@ffff8edaa4188900 x1680988519443328/t0(0) o104->oak-OST00f5@10.49.27.13@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1603126521 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Oct 19 09:55:21 oak-io5-s2 kernel: Lustre: 420261:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
Oct 19 09:55:23 oak-io5-s2 kernel: LustreError: 63791:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(4194304)  req@ffff8e6cbdc0f050 x1678198917287568/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:329/0 lens 488/448 e 0 to 0 dl 1603126559 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 09:55:23 oak-io5-s2 kernel: Lustre: oak-OST00cf: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 09:55:23 oak-io5-s2 kernel: Lustre: oak-OST00c1: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 09:55:23 oak-io5-s2 kernel: Lustre: Skipped 14 previous similar messages
Oct 19 09:55:23 oak-io5-s2 kernel: Lustre: Skipped 14 previous similar messages
Oct 19 09:55:23 oak-io5-s2 kernel: LustreError: 63791:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
Oct 19 09:55:25 oak-io5-s2 kernel: Lustre: 421278:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1603126514/real 1603126514]  req@ffff8edaa2719f80 x1680988519443264/t0(0) o104->oak-OST00f3@10.49.27.13@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1603126525 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Oct 19 09:55:25 oak-io5-s2 kernel: Lustre: 421278:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 56 previous similar messages
Oct 19 09:55:33 oak-io5-s2 kernel: Lustre: 421163:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1603126522/real 1603126522]  req@ffff8eda9b7d3600 x1680988519437568/t0(0) o104->oak-OST00d0@10.50.15.11@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1603126533 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Oct 19 09:55:33 oak-io5-s2 kernel: Lustre: 421163:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 132 previous similar messages
Oct 19 09:55:50 oak-io5-s2 kernel: Lustre: 416804:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1603126539/real 1603126539]  req@ffff8edaa418bf00 x1680988519436672/t0(0) o104->oak-OST00d6@10.50.15.7@o2ib2:15/16 lens 296/224 e 0 to 1 dl 1603126550 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Oct 19 09:55:50 oak-io5-s2 kernel: Lustre: 416804:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 209 previous similar messages
Oct 19 09:55:57 oak-io5-s2 kernel: LustreError: 62347:0:(ldlm_lib.c:3228:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff8e6cc5973850 x1654552973867600/t0(0) o3->7f0796a8-ed6e-138f-ae50-0463cb5b430a@10.210.12.119@tcp1:370/0 lens 488/440 e 0 to 0 dl 1603126600 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 09:55:57 oak-io5-s2 kernel: Lustre: oak-OST00e8: Bulk IO read error with 7f0796a8-ed6e-138f-ae50-0463cb5b430a (at 10.210.12.119@tcp1), client will retry: rc -107
Oct 19 09:55:58 oak-io5-s2 kernel: LustreError: 63814:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8e6cc10ad050 x1680089659386608/t0(0) o4->d3f31e49-709a-9027-c06d-e7875618ecd4@10.210.12.8@tcp1:370/0 lens 488/448 e 0 to 0 dl 1603126600 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 09:55:58 oak-io5-s2 kernel: Lustre: oak-OST00d7: Bulk IO write error with bf7f3100-aa36-e17d-473f-b840b304ff3e (at 10.210.12.31@tcp1), client will retry: rc = -110
Oct 19 09:55:58 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 09:55:58 oak-io5-s2 kernel: Lustre: oak-OST00e8: Bulk IO read error with 7f0796a8-ed6e-138f-ae50-0463cb5b430a (at 10.210.12.119@tcp1), client will retry: rc -110
Oct 19 09:55:58 oak-io5-s2 kernel: LustreError: 63814:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 16 previous similar messages
Oct 19 09:56:00 oak-io5-s2 kernel: LustreError: 63586:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff8e6ccf628850 x1674949589854624/t0(0) o3->6c0c453f-2604-6818-784d-e05fcb7efca9@10.210.12.129@tcp1:370/0 lens 488/440 e 0 to 0 dl 1603126600 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 09:56:00 oak-io5-s2 kernel: LustreError: 63586:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 3 previous similar messages
Oct 19 09:56:00 oak-io5-s2 kernel: Lustre: oak-OST00d3: Bulk IO read error with 6c0c453f-2604-6818-784d-e05fcb7efca9 (at 10.210.12.129@tcp1), client will retry: rc -110
Oct 19 09:56:00 oak-io5-s2 kernel: Lustre: Skipped 4 previous similar messages
Oct 19 09:56:00 oak-io5-s2 kernel: Lustre: oak-OST00f3: Bulk IO write error with 6c0c453f-2604-6818-784d-e05fcb7efca9 (at 10.210.12.129@tcp1), client will retry: rc = -110
Oct 19 09:56:00 oak-io5-s2 kernel: Lustre: Skipped 9 previous similar messages
Oct 19 09:56:14 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds
Oct 19 09:56:14 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Skipped 2 previous similar messages
Oct 19 09:56:14 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.2.217@o2ib5 (4): c: 0, oc: 0, rc: 8
Oct 19 09:56:14 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Skipped 2 previous similar messages
Oct 19 09:56:22 oak-io5-s2 kernel: Lustre: 416837:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1603126575/real 1603126575]  req@ffff8ed18f607980 x1680988519467520/t0(0) o106->oak-OST00f9@10.49.25.16@o2ib1:15/16 lens 296/280 e 0 to 1 dl 1603126582 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Oct 19 09:56:22 oak-io5-s2 kernel: Lustre: 416837:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 452 previous similar messages
Oct 19 09:56:51 oak-io5-s2 kernel: LustreError: 63793:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8e6cbaf0a050 x1678198929845808/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:424/0 lens 488/448 e 0 to 0 dl 1603126654 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 09:56:51 oak-io5-s2 kernel: LustreError: 63793:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 42 previous similar messages
Oct 19 09:56:51 oak-io5-s2 kernel: Lustre: oak-OST010d: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 09:56:51 oak-io5-s2 kernel: Lustre: Skipped 17 previous similar messages
Oct 19 09:57:27 oak-io5-s2 kernel: Lustre: 418258:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1603126640/real 1603126640]  req@ffff8edb04cf1200 x1680988519443136/t0(0) o104->oak-OST00f2@10.49.27.13@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1603126647 ref 1 fl Rpc:X/2/ffffffff rc 0/-1
Oct 19 09:57:27 oak-io5-s2 kernel: Lustre: 418258:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 904 previous similar messages
Oct 19 09:57:29 oak-io5-s2 kernel: LustreError: 416834:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.49.27.13@o2ib1) failed to reply to blocking AST (req@ffff8ed18f606c00 x1680988519441344 status 0 rc -5), evict it ns: filter-oak-OST00db_UUID lock: ffff8e6a6afa0b40/0xf1ba194985bb960 lrc: 4/0,0 mode: PR/PR res: [0x39c0000400:0x2609e4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 10.49.27.13@o2ib1 remote: 0x879a77bbe1bb6ec2 expref: 13 pid: 418371 timeout: 14498 lvb_type: 1
Oct 19 09:57:29 oak-io5-s2 kernel: LustreError: 138-a: oak-OST00d7: A client on nid 10.49.27.13@o2ib1 was evicted due to a lock blocking callback time out: rc -5
Oct 19 09:57:29 oak-io5-s2 kernel: LustreError: 413827:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 138s: evicting client at 10.49.27.13@o2ib1  ns: filter-oak-OST00d7_UUID lock: ffff8e6a7b7a18c0/0xf1ba194985ba71c lrc: 2/0,0 mode: --/PR res: [0x38c0000400:0x25c464:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x44801400010020 nid: 10.49.27.13@o2ib1 remote: 0x879a77bbe1bb647a expref: 8 pid: 64530 timeout: 0 lvb_type: 1
Oct 19 09:57:29 oak-io5-s2 kernel: LustreError: 420582:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.51.2.31@o2ib3: deadline 100:63s ago
                                     req@ffff8ecacdb02850 x1676696273028608/t0(0) o8-><?>@<?>:0/0 lens 520/0 e 0 to 0 dl 1603126586 ref 1 fl Interpret:/0/ffffffff rc 0/-1
Oct 19 09:57:29 oak-io5-s2 kernel: Lustre: 420582:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (100:63s); client may timeout.  req@ffff8ecacdb02850 x1676696273028608/t0(0) o8-><?>@<?>:0/0 lens 520/0 e 0 to 0 dl 1603126586 ref 1 fl Interpret:/0/ffffffff rc 0/-1
Oct 19 09:57:29 oak-io5-s2 kernel: Lustre: 420582:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 299 previous similar messages
Oct 19 09:57:29 oak-io5-s2 kernel: LustreError: 416834:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) Skipped 2 previous similar messages
Oct 19 09:57:40 oak-io5-s2 kernel: LustreError: 416804:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.50.15.7@o2ib2) failed to reply to blocking AST (req@ffff8edaa418bf00 x1680988519436672 status 0 rc -110), evict it ns: filter-oak-OST00d6_UUID lock: ffff8e6b0af07bc0/0xf1ba194985afa37 lrc: 4/0,0 mode: PR/PR res: [0xfbb04e:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400000020 nid: 10.50.15.7@o2ib2 remote: 0x52b6a4b99ee2a7b1 expref: 7 pid: 418374 timeout: 14503 lvb_type: 1
Oct 19 09:57:40 oak-io5-s2 kernel: LustreError: 138-a: oak-OST00d6: A client on nid 10.50.15.7@o2ib2 was evicted due to a lock blocking callback time out: rc -110
Oct 19 09:57:40 oak-io5-s2 kernel: LustreError: Skipped 2 previous similar messages
Oct 19 09:57:40 oak-io5-s2 kernel: LustreError: 413827:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 155s: evicting client at 10.50.15.7@o2ib2  ns: filter-oak-OST00d6_UUID lock: ffff8e6b0af07bc0/0xf1ba194985afa37 lrc: 3/0,0 mode: PR/PR res: [0xfbb04e:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400000020 nid: 10.50.15.7@o2ib2 remote: 0x52b6a4b99ee2a7b1 expref: 7 pid: 418374 timeout: 0 lvb_type: 1
Oct 19 09:57:46 oak-io5-s2 kernel: LustreError: 419474:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.50.15.7@o2ib2) failed to reply to blocking AST (req@ffff8edb0a1f7980 x1680988519437440 status 0 rc -110), evict it ns: filter-oak-OST00dd_UUID lock: ffff8ecd9ac1de80/0xf1ba194985b5bab lrc: 4/0,0 mode: PR/PR res: [0x24aa59:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400000020 nid: 10.50.15.7@o2ib2 remote: 0x52b6a4b99ee2a780 expref: 7 pid: 416753 timeout: 14509 lvb_type: 1
Oct 19 09:57:46 oak-io5-s2 kernel: LustreError: 138-a: oak-OST00f9: A client on nid 10.49.25.16@o2ib1 was evicted due to a lock blocking callback time out: rc -110
Oct 19 09:57:46 oak-io5-s2 kernel: LustreError: 413827:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.50.4.58@o2ib2  ns: filter-oak-OST0108_UUID lock: ffff8e6ccf29af40/0xf1ba19498595e9c lrc: 3/0,0 mode: PR/PR res: [0x4500000400:0x2c75f8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400000020 nid: 10.50.4.58@o2ib2 remote: 0x5a51af4e7b0db217 expref: 8 pid: 416799 timeout: 0 lvb_type: 1
Oct 19 09:57:46 oak-io5-s2 kernel: LustreError: 419474:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) Skipped 107 previous similar messages
Oct 19 09:57:48 oak-io5-s2 kernel: LustreError: 416763:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00dc: cli 548d000f-ab1a-9806-8901-4ccc0d3ca2ee claims 4218880 GRANT, real grant 2801664
Oct 19 09:57:48 oak-io5-s2 kernel: LustreError: 421378:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.49.27.13@o2ib1) failed to reply to blocking AST (req@ffff8edb0a1f3f00 x1680988519442880 status 0 rc -110), evict it ns: filter-oak-OST00f1_UUID lock: ffff8e6a6c359200/0xf1ba194985bb729 lrc: 4/0,0 mode: PR/PR res: [0x3f40000400:0x287084:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 10.49.27.13@o2ib1 remote: 0x879a77bbe1bb6c1b expref: 8 pid: 416800 timeout: 14512 lvb_type: 1
Oct 19 09:57:48 oak-io5-s2 kernel: LustreError: 138-a: oak-OST010a: A client on nid 10.49.27.13@o2ib1 was evicted due to a lock blocking callback time out: rc -110
Oct 19 09:57:48 oak-io5-s2 kernel: LustreError: Skipped 108 previous similar messages
Oct 19 09:57:48 oak-io5-s2 kernel: LustreError: 413827:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.49.27.13@o2ib1  ns: filter-oak-OST00f8_UUID lock: ffff8e6a774b8d80/0xf1ba194985baa09 lrc: 3/0,0 mode: PW/PW res: [0x4100000401:0x2a2204:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->8191) flags: 0x60000400030020 nid: 10.49.27.13@o2ib1 remote: 0x879a77bbe1bb665d expref: 13 pid: 422594 timeout: 0 lvb_type: 0
Oct 19 09:57:48 oak-io5-s2 kernel: LustreError: 413827:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 52 previous similar messages
Oct 19 09:57:48 oak-io5-s2 kernel: LustreError: 421378:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) Skipped 69 previous similar messages
Oct 19 09:57:50 oak-io5-s2 kernel: LustreError: 416761:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00dc: cli 548d000f-ab1a-9806-8901-4ccc0d3ca2ee claims 4218880 GRANT, real grant 0
Oct 19 09:57:51 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds
Oct 19 09:57:51 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.2.217@o2ib5 (3): c: 0, oc: 0, rc: 8
Oct 19 09:57:52 oak-io5-s2 kernel: LustreError: 453748:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00dc: cli 548d000f-ab1a-9806-8901-4ccc0d3ca2ee claims 4218880 GRANT, real grant 0
Oct 19 09:57:54 oak-io5-s2 kernel: LustreError: 420927:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.49.27.13@o2ib1) failed to reply to blocking AST (req@ffff8eda953c0000 x1680988519467072 status 0 rc -110), evict it ns: filter-oak-OST00ce_UUID lock: ffff8e6cd5fb4800/0xf1ba194985ba8ff lrc: 4/0,0 mode: PR/PR res: [0x3680000400:0x216464:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 10.49.27.13@o2ib1 remote: 0x879a77bbe1bb65b5 expref: 10 pid: 417502 timeout: 14517 lvb_type: 1
Oct 19 09:57:54 oak-io5-s2 kernel: LustreError: 138-a: oak-OST010c: A client on nid 10.49.27.13@o2ib1 was evicted due to a lock blocking callback time out: rc -110
Oct 19 09:57:54 oak-io5-s2 kernel: LustreError: Skipped 69 previous similar messages
Oct 19 09:57:54 oak-io5-s2 kernel: LustreError: 413827:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.49.27.13@o2ib1  ns: filter-oak-OST010c_UUID lock: ffff8e6a7c3521c0/0xf1ba194985ba811 lrc: 3/0,0 mode: PR/PR res: [0x4600000401:0x2b5ec4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 10.49.27.13@o2ib1 remote: 0x879a77bbe1bb6529 expref: 11 pid: 420263 timeout: 0 lvb_type: 1
Oct 19 09:57:54 oak-io5-s2 kernel: LustreError: 413827:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 41 previous similar messages
Oct 19 09:57:54 oak-io5-s2 kernel: LustreError: 420927:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) Skipped 12 previous similar messages
Oct 19 09:57:54 oak-io5-s2 kernel: LustreError: 403:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00dc: cli 548d000f-ab1a-9806-8901-4ccc0d3ca2ee claims 4218880 GRANT, real grant 0
Oct 19 09:58:00 oak-io5-s2 kernel: LustreError: 453736:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.49.27.13@o2ib1 arrived at 1603126680 with bad export cookie 1088641394522596993
Oct 19 09:58:00 oak-io5-s2 kernel: LustreError: 453736:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 8 previous similar messages
Oct 19 09:58:01 oak-io5-s2 kernel: LustreError: 453736:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.50.4.58@o2ib2 arrived at 1603126681 with bad export cookie 1088641394522297708
Oct 19 09:58:01 oak-io5-s2 kernel: LustreError: 453736:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 20 previous similar messages
Oct 19 09:58:02 oak-io5-s2 kernel: LustreError: 402:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff8ecdd7ba3050 x1672215846222016/t0(0) o3->3f535fbe-c6c8-4@10.50.6.41@o2ib2:494/0 lens 504/440 e 0 to 0 dl 1603126724 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 09:58:02 oak-io5-s2 kernel: Lustre: oak-OST00d2: Bulk IO write error with a202f1b7-92bd-4 (at 10.50.4.59@o2ib2), client will retry: rc = -110
Oct 19 09:58:02 oak-io5-s2 kernel: Lustre: Skipped 7 previous similar messages
Oct 19 09:58:02 oak-io5-s2 kernel: Lustre: oak-OST010e: Bulk IO read error with 3f535fbe-c6c8-4 (at 10.50.6.41@o2ib2), client will retry: rc -110
Oct 19 09:58:02 oak-io5-s2 kernel: Lustre: Skipped 21 previous similar messages
Oct 19 09:58:02 oak-io5-s2 kernel: LustreError: 402:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 12 previous similar messages
Oct 19 09:58:05 oak-io5-s2 kernel: LustreError: 63790:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(1516320)  req@ffff8e760c0e6850 x1672252535017216/t0(0) o4->4971349b-bf83-4@10.49.22.12@o2ib1:493/0 lens 488/448 e 0 to 0 dl 1603126723 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 09:58:05 oak-io5-s2 kernel: LustreError: 63789:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 0(4194304)  req@ffff8e760c0e3050 x1672467345716288/t0(0) o3->159d5082-a767-4@10.49.27.21@o2ib1:493/0 lens 488/440 e 0 to 0 dl 1603126723 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 09:58:05 oak-io5-s2 kernel: LustreError: 63790:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
Oct 19 09:58:06 oak-io5-s2 kernel: Lustre: oak-OST00ee: Export ffff8ec267aa6c00 already connecting from 10.51.0.18@o2ib3
Oct 19 09:58:08 oak-io5-s2 kernel: LustreError: 401:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(1048576)  req@ffff8e6cc9232050 x1672398639254720/t0(0) o4->a202f1b7-92bd-4@10.50.4.59@o2ib2:494/0 lens 488/448 e 0 to 0 dl 1603126724 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 09:58:08 oak-io5-s2 kernel: LustreError: 63808:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 0(4194304)  req@ffff8e6cd2b95050 x1672467345716416/t0(0) o3->159d5082-a767-4@10.49.27.21@o2ib1:495/0 lens 488/440 e 0 to 0 dl 1603126725 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 09:58:08 oak-io5-s2 kernel: LustreError: 63808:0:(ldlm_lib.c:3294:target_bulk_io()) Skipped 5 previous similar messages
Oct 19 09:58:08 oak-io5-s2 kernel: Lustre: oak-OST00eb: Bulk IO read error with 791c5260-47dd-4 (at 10.49.0.61@o2ib1), client will retry: rc -110
Oct 19 09:58:08 oak-io5-s2 kernel: Lustre: Skipped 36 previous similar messages
Oct 19 09:58:08 oak-io5-s2 kernel: Lustre: oak-OST00f3: Export ffff8e6cdaf66400 already connecting from 10.51.0.18@o2ib3
Oct 19 09:58:08 oak-io5-s2 kernel: LustreError: 401:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 6 previous similar messages
Oct 19 09:58:13 oak-io5-s2 kernel: LustreError: 28363:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8ecc4432d800
Oct 19 09:58:13 oak-io5-s2 kernel: LustreError: 28357:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8ecc4432d800
Oct 19 09:58:13 oak-io5-s2 kernel: LustreError: 28358:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8ecc4432d800
Oct 19 09:58:13 oak-io5-s2 kernel: LustreError: 28363:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8ec26845d400
Oct 19 09:58:13 oak-io5-s2 kernel: LustreError: 64581:0:(ldlm_lib.c:3285:target_bulk_io()) @@@ network error on bulk READ  req@ffff8ed929ce8050 x1673216917878272/t0(0) o3->26503748-f849-4@10.50.12.3@o2ib2:496/0 lens 488/440 e 0 to 0 dl 1603126726 ref 1 fl Interpret:/2/0 rc 0/0
Oct 19 09:58:13 oak-io5-s2 kernel: LustreError: 28357:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8ec26845d400
Oct 19 09:58:13 oak-io5-s2 kernel: LustreError: 28358:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8ecdd8a76c00
Oct 19 09:58:13 oak-io5-s2 kernel: LustreError: 28356:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8ecdd8a76c00
Oct 19 09:58:13 oak-io5-s2 kernel: LustreError: 28363:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8e775380e400
Oct 19 09:58:13 oak-io5-s2 kernel: LustreError: 28357:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8ecdcda2d800
Oct 19 09:58:13 oak-io5-s2 kernel: LustreError: 28358:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8ecdcda2d800
Oct 19 09:58:26 oak-io5-s2 kernel: LustreError: 63809:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00dc: cli 548d000f-ab1a-9806-8901-4ccc0d3ca2ee claims 4218880 GRANT, real grant 3178496
Oct 19 09:58:34 oak-io5-s2 kernel: Lustre: oak-OST00c0: Client 2850c699-04d9-4 (at 10.50.1.30@o2ib2) reconnecting
Oct 19 09:58:34 oak-io5-s2 kernel: Lustre: Skipped 112217 previous similar messages
Oct 19 09:59:03 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8ed488682c00
Oct 19 09:59:03 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8ed488682c00
Oct 19 09:59:03 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8eccb1b40000
Oct 19 09:59:03 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8eccb1b40000
Oct 19 09:59:06 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8eccc17e0c00
Oct 19 09:59:06 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8ec267913000
Oct 19 09:59:06 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8edaffbfe400
Oct 19 09:59:08 oak-io5-s2 kernel: LustreError: 407:0:(ldlm_lib.c:3285:target_bulk_io()) @@@ network error on bulk READ  req@ffff8ecdd0448850 x1675506759540416/t0(0) o3->7fb146e2-bb02-4@10.51.1.30@o2ib3:554/0 lens 488/440 e 0 to 0 dl 1603126784 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 09:59:08 oak-io5-s2 kernel: Lustre: oak-OST00c6: Bulk IO read error with 7fb146e2-bb02-4 (at 10.51.1.30@o2ib3), client will retry: rc -110
Oct 19 09:59:08 oak-io5-s2 kernel: Lustre: oak-OST00c6: Bulk IO read error with 7fb146e2-bb02-4 (at 10.51.1.30@o2ib3), client will retry: rc -110
Oct 19 09:59:08 oak-io5-s2 kernel: Lustre: Skipped 10 previous similar messages
Oct 19 09:59:08 oak-io5-s2 kernel: Lustre: Skipped 10 previous similar messages
Oct 19 09:59:08 oak-io5-s2 kernel: LustreError: 407:0:(ldlm_lib.c:3285:target_bulk_io()) Skipped 8 previous similar messages
Oct 19 09:59:59 oak-io5-s2 kernel: Lustre: oak-OST010b: Export ffff8e6ce8be0400 already connecting from 10.51.2.32@o2ib3
Oct 19 09:59:59 oak-io5-s2 kernel: Lustre: Skipped 3 previous similar messages
Oct 19 10:00:01 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds
Oct 19 10:00:01 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Skipped 11 previous similar messages
Oct 19 10:00:01 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.2.217@o2ib5 (1): c: 0, oc: 2, rc: 8
Oct 19 10:00:01 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Skipped 11 previous similar messages
Oct 19 10:00:04 oak-io5-s2 kernel: LNet: 28355:0:(o2iblnd_cb.c:1511:kiblnd_reconnect_peer()) Abort reconnection of 10.0.2.217@o2ib5: accepting
Oct 19 10:00:17 oak-io5-s2 kernel: Lustre: oak-OST00c0: Export ffff8e6cc9fcc400 already connecting from 10.51.1.42@o2ib3
Oct 19 10:00:17 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 10:00:29 oak-io5-s2 kernel: Lustre: oak-OST00fa: Export ffff8ebe8f03f800 already connecting from 10.51.12.9@o2ib3
Oct 19 10:00:29 oak-io5-s2 kernel: Lustre: Skipped 3 previous similar messages
Oct 19 10:00:42 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8ebe23acec00
Oct 19 10:00:45 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8ebe26196800
Oct 19 10:00:45 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8ebdca608000
Oct 19 10:00:45 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8ebdca608000
Oct 19 10:00:45 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8ebe26194800
Oct 19 10:00:45 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8ec23f003c00
Oct 19 10:00:45 oak-io5-s2 kernel: LustreError: 63798:0:(ldlm_lib.c:3285:target_bulk_io()) @@@ network error on bulk READ  req@ffff8ebf237e2050 x1675110492550016/t0(0) o3->2df1f7fc-7a05-4@10.51.12.19@o2ib3:652/0 lens 488/440 e 0 to 0 dl 1603126882 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:00:45 oak-io5-s2 kernel: Lustre: oak-OST00c6: Bulk IO read error with 2df1f7fc-7a05-4 (at 10.51.12.19@o2ib3), client will retry: rc -110
Oct 19 10:00:45 oak-io5-s2 kernel: Lustre: Skipped 3 previous similar messages
Oct 19 10:00:45 oak-io5-s2 kernel: LustreError: 63798:0:(ldlm_lib.c:3285:target_bulk_io()) Skipped 2 previous similar messages
Oct 19 10:00:51 oak-io5-s2 kernel: LustreError: 28365:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8ebdac330000
Oct 19 10:00:51 oak-io5-s2 kernel: LustreError: 28367:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8ebe279fb400
Oct 19 10:00:51 oak-io5-s2 kernel: LustreError: 391989:0:(ldlm_lib.c:3285:target_bulk_io()) @@@ network error on bulk READ  req@ffff8ebf237f2850 x1675110492550400/t0(0) o3->2df1f7fc-7a05-4@10.51.12.19@o2ib3:652/0 lens 488/440 e 0 to 0 dl 1603126882 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:00:51 oak-io5-s2 kernel: LustreError: 28366:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8ebe279fb400
Oct 19 10:00:51 oak-io5-s2 kernel: LustreError: 28365:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8ec23f003c00
Oct 19 10:00:51 oak-io5-s2 kernel: LustreError: 28367:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8ebe23acec00
Oct 19 10:00:53 oak-io5-s2 kernel: Lustre: oak-OST00ce: Export ffff8ebe2e84cc00 already connecting from 10.51.1.43@o2ib3
Oct 19 10:00:53 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 10:01:17 oak-io5-s2 kernel: Lustre: 420087:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (100:1s); client may timeout.  req@ffff8ebf5e44d850 x1679863656515328/t0(0) o8->79a83f3f-8d8d-4@10.51.1.42@o2ib3:0/0 lens 520/384 e 0 to 0 dl 1603126876 ref 1 fl Complete:/0/0 rc 0/0
Oct 19 10:01:17 oak-io5-s2 kernel: Lustre: 420087:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 1097 previous similar messages
Oct 19 10:01:29 oak-io5-s2 kernel: Lustre: oak-OST00ed: Export ffff8ebd5559c400 already connecting from 10.51.1.19@o2ib3
Oct 19 10:01:29 oak-io5-s2 kernel: Lustre: Skipped 6 previous similar messages
Oct 19 10:01:31 oak-io5-s2 kernel: LNetError: 28597:0:(lib-move.c:1999:lnet_handle_find_routed_path()) no route to 10.51.1.21@o2ib3 from <?>
Oct 19 10:01:35 oak-io5-s2 kernel: Lustre: oak-OST00dc: Client 3901acb9-1d06-4 (at 10.51.13.19@o2ib3) refused connection, still busy with 6 references
Oct 19 10:01:35 oak-io5-s2 kernel: Lustre: Skipped 93 previous similar messages
Oct 19 10:03:47 oak-io5-s2 kernel: Lustre: oak-OST010a: Connection restored to 35dd6bf7-b09b-4 (at 10.51.12.2@o2ib3)
Oct 19 10:03:47 oak-io5-s2 kernel: Lustre: Skipped 165320 previous similar messages
Oct 19 10:04:57 oak-io5-s2 kernel: Lustre: oak-OST010b: haven't heard from client f9424431-ef38-4 (at 10.51.2.1@o2ib3) in 321 seconds. I think it's dead, and I am evicting it. exp ffff8eccaf78fc00, cur 1603127097 expire 1603126947 last 1603126776
Oct 19 10:04:57 oak-io5-s2 kernel: Lustre: Skipped 14982 previous similar messages
Oct 19 10:09:18 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds
Oct 19 10:09:18 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Skipped 16 previous similar messages
Oct 19 10:09:18 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.2.216@o2ib5 (0): c: 3, oc: 0, rc: 3
Oct 19 10:09:18 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Skipped 16 previous similar messages
Oct 19 10:09:18 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8ebe812a3400
Oct 19 10:09:18 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8ebc13d4f800
Oct 19 10:09:18 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8ebc13d4f800
Oct 19 10:09:18 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8ebc15b3dc00
Oct 19 10:09:18 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8ebc15b3dc00
Oct 19 10:09:22 oak-io5-s2 kernel: LustreError: 453751:0:(ldlm_lib.c:3285:target_bulk_io()) @@@ network error on bulk READ  req@ffff8ebce388a050 x1675110493121664/t0(0) o3->2df1f7fc-7a05-4@10.51.12.19@o2ib3:404/0 lens 488/440 e 1 to 0 dl 1603127389 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:09:22 oak-io5-s2 kernel: Lustre: oak-OST00ff: Bulk IO read error with 2df1f7fc-7a05-4 (at 10.51.12.19@o2ib3), client will retry: rc -110
Oct 19 10:09:22 oak-io5-s2 kernel: Lustre: Skipped 6 previous similar messages
Oct 19 10:09:22 oak-io5-s2 kernel: LustreError: 453751:0:(ldlm_lib.c:3285:target_bulk_io()) Skipped 5 previous similar messages
Oct 19 10:09:52 oak-io5-s2 kernel: Lustre: oak-OST00ff: Client 2df1f7fc-7a05-4 (at 10.51.12.19@o2ib3) reconnecting
Oct 19 10:09:52 oak-io5-s2 kernel: Lustre: Skipped 32183 previous similar messages
Oct 19 10:14:15 oak-io5-s2 kernel: LustreError: 64587:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00dc: cli 548d000f-ab1a-9806-8901-4ccc0d3ca2ee claims 4218880 GRANT, real grant 3407872
Oct 19 10:15:14 oak-io5-s2 kernel: Lustre: oak-OST00f6: haven't heard from client 8cb0b8ed-41dd-4 (at 10.49.8.17@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e705dac4800, cur 1603127714 expire 1603127564 last 1603127487
Oct 19 10:15:14 oak-io5-s2 kernel: Lustre: Skipped 1872 previous similar messages
Oct 19 10:26:59 oak-io5-s2 kernel: Lustre: oak-OST00c1: Connection restored to  (at 10.0.2.221@o2ib5)
Oct 19 10:26:59 oak-io5-s2 kernel: Lustre: Skipped 63 previous similar messages
Oct 19 10:28:32 oak-io5-s2 kernel: Lustre: oak-OST00c1: Connection restored to b923a102-900b-4 (at 10.51.2.21@o2ib3)
Oct 19 10:28:32 oak-io5-s2 kernel: Lustre: Skipped 1353 previous similar messages
Oct 19 10:37:55 oak-io5-s2 kernel: Lustre: oak-OST00c2: Client 5d6c6b8b-ad90-4 (at 10.50.6.54@o2ib2) reconnecting
Oct 19 10:37:55 oak-io5-s2 kernel: Lustre: oak-OST00d5: Connection restored to  (at 10.50.6.54@o2ib2)
Oct 19 10:37:55 oak-io5-s2 kernel: Lustre: Skipped 275 previous similar messages
Oct 19 10:37:55 oak-io5-s2 kernel: Lustre: Skipped 9 previous similar messages
Oct 19 10:38:11 oak-io5-s2 kernel: LustreError: 63812:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8ebe99e14850 x1678193443771248/t0(0) o4->509402fa-31a4-b854-467a-09186a1277b2@10.210.12.21@tcp1:601/0 lens 488/448 e 0 to 0 dl 1603129096 ref 2 fl Interpret:/0/0 rc 0/0
Oct 19 10:38:11 oak-io5-s2 kernel: Lustre: oak-OST00c0: Bulk IO write error with 509402fa-31a4-b854-467a-09186a1277b2 (at 10.210.12.21@tcp1), client will retry: rc = -110
Oct 19 10:38:11 oak-io5-s2 kernel: Lustre: Skipped 18 previous similar messages
Oct 19 10:38:11 oak-io5-s2 kernel: LustreError: 63812:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 27 previous similar messages
Oct 19 10:38:16 oak-io5-s2 kernel: LustreError: 410:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 0(4194304)  req@ffff8e6ce2a1f850 x1677104239188608/t0(0) o3->89be6b1d-933c-4@10.50.10.1@o2ib2:621/0 lens 488/440 e 1 to 0 dl 1603129116 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:38:16 oak-io5-s2 kernel: Lustre: oak-OST00e6: Bulk IO read error with 89be6b1d-933c-4 (at 10.50.10.1@o2ib2), client will retry: rc -110
Oct 19 10:38:16 oak-io5-s2 kernel: Lustre: Skipped 2 previous similar messages
Oct 19 10:38:16 oak-io5-s2 kernel: LustreError: 410:0:(ldlm_lib.c:3294:target_bulk_io()) Skipped 13 previous similar messages
Oct 19 10:39:11 oak-io5-s2 kernel: Lustre: oak-OST00e6: Client fea0435a-aefb-4 (at 10.50.8.41@o2ib2) reconnecting
Oct 19 10:39:11 oak-io5-s2 kernel: Lustre: Skipped 11490 previous similar messages
Oct 19 10:40:48 oak-io5-s2 kernel: Lustre: oak-OST00f5: haven't heard from client 00d5a382-9e9d-4 (at 10.50.6.20@o2ib2) in 225 seconds. I think it's dead, and I am evicting it. exp ffff8e6cff153000, cur 1603129248 expire 1603129098 last 1603129023
Oct 19 10:40:48 oak-io5-s2 kernel: Lustre: Skipped 176 previous similar messages
Oct 19 10:41:25 oak-io5-s2 kernel: Lustre: oak-OST00d7: Export ffff8e619c21f400 already connecting from 10.49.29.6@o2ib1
Oct 19 10:41:25 oak-io5-s2 kernel: Lustre: Skipped 10 previous similar messages
Oct 19 10:41:41 oak-io5-s2 kernel: Lustre: oak-OST00c3: Client 65e78a10-f94c-4 (at 10.49.20.7@o2ib1) reconnecting
Oct 19 10:41:41 oak-io5-s2 kernel: Lustre: Skipped 84569 previous similar messages
Oct 19 10:42:04 oak-io5-s2 kernel: Lustre: oak-OST00ca: haven't heard from client f345b700-ff28-4 (at 10.50.1.43@o2ib2) in 218 seconds. I think it's dead, and I am evicting it. exp ffff8e775d157400, cur 1603129324 expire 1603129174 last 1603129106
Oct 19 10:42:04 oak-io5-s2 kernel: Lustre: Skipped 62658 previous similar messages
Oct 19 10:42:57 oak-io5-s2 kernel: Lustre: oak-OST00c6: Connection restored to 4e513f37-035a-4 (at 10.51.2.11@o2ib3)
Oct 19 10:42:57 oak-io5-s2 kernel: Lustre: oak-OST00c2: Connection restored to 4e513f37-035a-4 (at 10.51.2.11@o2ib3)
Oct 19 10:42:57 oak-io5-s2 kernel: Lustre: Skipped 212796 previous similar messages
Oct 19 10:45:12 oak-io5-s2 kernel: Lustre: oak-OST00cc: haven't heard from client e26b04e0-563b-4 (at 10.50.7.69@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7a3fa76400, cur 1603129512 expire 1603129362 last 1603129285
Oct 19 10:45:12 oak-io5-s2 kernel: Lustre: Skipped 77 previous similar messages
Oct 19 10:54:39 oak-io5-s2 kernel: Lustre: oak-OST00f2: Connection restored to  (at 10.50.7.69@o2ib2)
Oct 19 10:54:39 oak-io5-s2 kernel: Lustre: Skipped 139 previous similar messages
Oct 19 10:55:15 oak-io5-s2 kernel: Lustre: oak-OST00cc: Client 73cc681c-0337-4 (at 10.50.5.63@o2ib2) reconnecting
Oct 19 10:55:15 oak-io5-s2 kernel: Lustre: Skipped 55124 previous similar messages
Oct 19 10:55:37 oak-io5-s2 kernel: LustreError: 347874:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8e2ee05c4050 x1669823930818128/t0(0) o4->bf7f3100-aa36-e17d-473f-b840b304ff3e@10.210.12.31@tcp1:159/0 lens 488/448 e 1 to 0 dl 1603130164 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:55:37 oak-io5-s2 kernel: Lustre: oak-OST00fb: Bulk IO write error with bf7f3100-aa36-e17d-473f-b840b304ff3e (at 10.210.12.31@tcp1), client will retry: rc = -110
Oct 19 10:55:37 oak-io5-s2 kernel: Lustre: Skipped 3 previous similar messages
Oct 19 10:55:38 oak-io5-s2 kernel: LustreError: 347866:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8e2f97597050 x1668624343479104/t0(0) o4->c5a17f7e-ff08-fcd8-c481-e40cd9115c51@10.210.12.127@tcp1:162/0 lens 488/448 e 1 to 0 dl 1603130167 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:55:38 oak-io5-s2 kernel: LustreError: 347866:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 2 previous similar messages
Oct 19 10:55:38 oak-io5-s2 kernel: Lustre: oak-OST0106: Bulk IO write error with c5a17f7e-ff08-fcd8-c481-e40cd9115c51 (at 10.210.12.127@tcp1), client will retry: rc = -110
Oct 19 10:55:39 oak-io5-s2 kernel: Lustre: oak-OST00cb: Bulk IO read error with e8da4851-25c4-4 (at 10.50.9.48@o2ib2), client will retry: rc -110
Oct 19 10:55:39 oak-io5-s2 kernel: Lustre: Skipped 8 previous similar messages
Oct 19 10:55:39 oak-io5-s2 kernel: LustreError: 347848:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff8ebda45cc050 x1672177687188160/t0(0) o3->e8da4851-25c4-4@10.50.9.48@o2ib2:157/0 lens 488/440 e 1 to 0 dl 1603130162 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:55:39 oak-io5-s2 kernel: LustreError: 347848:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 3 previous similar messages
Oct 19 10:55:42 oak-io5-s2 kernel: LustreError: 416760:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8e7a4e926050 x1669823930818288/t0(0) o4->bf7f3100-aa36-e17d-473f-b840b304ff3e@10.210.12.31@tcp1:163/0 lens 488/448 e 1 to 0 dl 1603130168 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:55:42 oak-io5-s2 kernel: Lustre: oak-OST00fb: Bulk IO write error with bf7f3100-aa36-e17d-473f-b840b304ff3e (at 10.210.12.31@tcp1), client will retry: rc = -110
Oct 19 10:55:42 oak-io5-s2 kernel: Lustre: Skipped 2 previous similar messages
Oct 19 10:55:42 oak-io5-s2 kernel: LustreError: 416760:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 2 previous similar messages
Oct 19 10:55:46 oak-io5-s2 kernel: LustreError: 347860:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff8edadef65050 x1674447861840272/t0(0) o4->1992388e-dcdd-33fa-f7c0-7602fbc70216@10.210.12.81@tcp1:160/0 lens 488/448 e 1 to 0 dl 1603130165 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:55:46 oak-io5-s2 kernel: Lustre: oak-OST00e5: Bulk IO write error with 2675d9e5-c3be-60e2-3a72-ea55143ec798 (at 10.210.12.25@tcp1), client will retry: rc = -110
Oct 19 10:55:46 oak-io5-s2 kernel: Lustre: Skipped 5 previous similar messages
Oct 19 10:55:46 oak-io5-s2 kernel: LustreError: 347860:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
Oct 19 10:55:47 oak-io5-s2 kernel: LustreError: 410:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8e2ee97a8850 x1669823930820032/t0(0) o4->bf7f3100-aa36-e17d-473f-b840b304ff3e@10.210.12.31@tcp1:181/0 lens 488/448 e 0 to 0 dl 1603130186 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:55:47 oak-io5-s2 kernel: LustreError: 410:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 3 previous similar messages
Oct 19 10:55:50 oak-io5-s2 kernel: Lustre: oak-OST0106: Bulk IO write error with c5a17f7e-ff08-fcd8-c481-e40cd9115c51 (at 10.210.12.127@tcp1), client will retry: rc = -110
Oct 19 10:55:50 oak-io5-s2 kernel: Lustre: Skipped 6 previous similar messages
Oct 19 10:55:51 oak-io5-s2 kernel: LustreError: 347863:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff8ed4918b8850 x1674447861840432/t0(0) o4->1992388e-dcdd-33fa-f7c0-7602fbc70216@10.210.12.81@tcp1:162/0 lens 488/448 e 1 to 0 dl 1603130167 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:55:51 oak-io5-s2 kernel: LustreError: 347862:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff8ebda45c9850 x1672381856197632/t0(0) o4->61c42114-1095-4@10.50.14.5@o2ib2:162/0 lens 488/448 e 1 to 0 dl 1603130167 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:55:51 oak-io5-s2 kernel: LustreError: 347812:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff8ebe96e39850 x1672381856197760/t0(0) o4->61c42114-1095-4@10.50.14.5@o2ib2:162/0 lens 488/448 e 1 to 0 dl 1603130167 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:55:51 oak-io5-s2 kernel: LustreError: 347863:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
Oct 19 10:56:01 oak-io5-s2 kernel: LustreError: 67802:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff8e2ee05c7050 x1669823930818128/t0(0) o4->bf7f3100-aa36-e17d-473f-b840b304ff3e@10.210.12.31@tcp1:173/0 lens 488/448 e 0 to 0 dl 1603130178 ref 1 fl Interpret:/2/0 rc 0/0
Oct 19 10:56:01 oak-io5-s2 kernel: LustreError: 347872:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff8e2ee97af050 x1669688660158608/t0(0) o3->f73489af-edd7-0720-ca37-908500000b7c@10.210.12.131@tcp1:185/0 lens 488/440 e 0 to 0 dl 1603130190 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:56:01 oak-io5-s2 kernel: Lustre: oak-OST0108: Bulk IO read error with f73489af-edd7-0720-ca37-908500000b7c (at 10.210.12.131@tcp1), client will retry: rc -110
Oct 19 10:56:01 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 10:56:01 oak-io5-s2 kernel: Lustre: oak-OST00d7: Bulk IO write error with bf7f3100-aa36-e17d-473f-b840b304ff3e (at 10.210.12.31@tcp1), client will retry: rc = -110
Oct 19 10:56:01 oak-io5-s2 kernel: Lustre: Skipped 4 previous similar messages
Oct 19 10:56:01 oak-io5-s2 kernel: LustreError: 67802:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 6 previous similar messages
Oct 19 10:56:10 oak-io5-s2 kernel: LustreError: 347828:0:(ldlm_lib.c:3228:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff8ed017e38850 x1674447861840576/t0(0) o4->1992388e-dcdd-33fa-f7c0-7602fbc70216@10.210.12.81@tcp1:171/0 lens 488/448 e 0 to 0 dl 1603130176 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:56:10 oak-io5-s2 kernel: LustreError: 347828:0:(ldlm_lib.c:3228:target_bulk_io()) Skipped 2 previous similar messages
Oct 19 10:56:11 oak-io5-s2 kernel: LustreError: 347812:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8ecd5d435850 x1654553630018976/t0(0) o4->2675d9e5-c3be-60e2-3a72-ea55143ec798@10.210.12.25@tcp1:171/0 lens 488/448 e 0 to 0 dl 1603130176 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:56:11 oak-io5-s2 kernel: LustreError: 347812:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 6 previous similar messages
Oct 19 10:56:17 oak-io5-s2 kernel: Lustre: oak-OST010b: Bulk IO write error with d3f31e49-709a-9027-c06d-e7875618ecd4 (at 10.210.12.8@tcp1), client will retry: rc = -110
Oct 19 10:56:17 oak-io5-s2 kernel: Lustre: Skipped 20 previous similar messages
Oct 19 10:56:31 oak-io5-s2 kernel: LustreError: 347888:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(1291284)  req@ffff8e29fbfa6850 x1678048447645952/t0(0) o4->19e1ae8f-b27a-4@10.50.12.14@o2ib2:206/0 lens 488/448 e 1 to 0 dl 1603130211 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:57:21 oak-io5-s2 kernel: LustreError: 347828:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 0(4194304)  req@ffff8ed9256fb050 x1677421553383040/t0(0) o3->f6a70503-f583-4@10.51.2.20@o2ib3:267/0 lens 488/440 e 0 to 0 dl 1603130272 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:57:21 oak-io5-s2 kernel: Lustre: oak-OST0105: Bulk IO read error with f6a70503-f583-4 (at 10.51.2.20@o2ib3), client will retry: rc -110
Oct 19 10:57:21 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 10:57:21 oak-io5-s2 kernel: LustreError: 347828:0:(ldlm_lib.c:3294:target_bulk_io()) Skipped 3 previous similar messages
Oct 19 10:58:10 oak-io5-s2 kernel: LustreError: 121388:0:(ldlm_lib.c:3228:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff8e29efcbe050 x1672840409489216/t0(0) o3->6d7b37a5-85dd-4@10.49.0.62@o2ib1:328/0 lens 488/440 e 0 to 0 dl 1603130333 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:58:10 oak-io5-s2 kernel: Lustre: oak-OST00d2: Bulk IO read error with 6d7b37a5-85dd-4 (at 10.49.0.62@o2ib1), client will retry: rc -107
Oct 19 10:58:10 oak-io5-s2 kernel: Lustre: Skipped 3 previous similar messages
Oct 19 10:58:10 oak-io5-s2 kernel: LustreError: 121388:0:(ldlm_lib.c:3228:target_bulk_io()) Skipped 1 previous similar message
Oct 19 10:58:10 oak-io5-s2 kernel: LustreError: 347861:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8ed4918bc050 x1672381856220096/t0(0) o4->61c42114-1095-4@10.50.14.5@o2ib2:327/0 lens 488/448 e 0 to 0 dl 1603130332 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 10:58:10 oak-io5-s2 kernel: LustreError: 347861:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 17 previous similar messages
Oct 19 10:58:10 oak-io5-s2 kernel: Lustre: oak-OST00cd: Bulk IO write error with 61c42114-1095-4 (at 10.50.14.5@o2ib2), client will retry: rc = -110
Oct 19 10:58:10 oak-io5-s2 kernel: Lustre: Skipped 9 previous similar messages
Oct 19 10:58:14 oak-io5-s2 kernel: Lustre: oak-OST00cc: haven't heard from client b7bb3f82-daa1-4 (at 10.50.2.8@o2ib2) in 221 seconds. I think it's dead, and I am evicting it. exp ffff8ebf5ec80400, cur 1603130294 expire 1603130144 last 1603130073
Oct 19 10:58:14 oak-io5-s2 kernel: Lustre: Skipped 34 previous similar messages
Oct 19 11:08:48 oak-io5-s2 kernel: Lustre: oak-OST00d0: Client 48ce6b22-151b-4 (at 10.49.27.10@o2ib1) reconnecting
Oct 19 11:08:48 oak-io5-s2 kernel: Lustre: oak-OST00d1: Connection restored to  (at 10.49.27.10@o2ib1)
Oct 19 11:08:48 oak-io5-s2 kernel: Lustre: Skipped 181143 previous similar messages
Oct 19 11:08:48 oak-io5-s2 kernel: Lustre: Skipped 163606 previous similar messages
Oct 19 11:08:49 oak-io5-s2 kernel: LustreError: 347805:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8ed7201ce050 x1654591516725264/t0(0) o4->83fe1960-46d3-ce9b-0b3e-87902c6e6371@10.210.12.27@tcp1:207/0 lens 488/448 e 0 to 0 dl 1603130967 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 11:08:49 oak-io5-s2 kernel: LustreError: 347805:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 19 previous similar messages
Oct 19 11:08:49 oak-io5-s2 kernel: Lustre: oak-OST00fc: Bulk IO write error with 83fe1960-46d3-ce9b-0b3e-87902c6e6371 (at 10.210.12.27@tcp1), client will retry: rc = -110
Oct 19 11:08:49 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 11:09:06 oak-io5-s2 kernel: LustreError: 347860:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1048576(1291284)  req@ffff8ecdd251c850 x1678145837443712/t0(0) o4->cc25f015-1657-4@10.50.15.8@o2ib2:218/0 lens 488/448 e 0 to 0 dl 1603130978 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 11:09:16 oak-io5-s2 kernel: LustreError: 416763:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff8ecdc68f3850 x1654591516725232/t0(0) o4->83fe1960-46d3-ce9b-0b3e-87902c6e6371@10.210.12.27@tcp1:223/0 lens 488/448 e 0 to 0 dl 1603130983 ref 1 fl Interpret:/2/0 rc 0/0
Oct 19 11:09:26 oak-io5-s2 kernel: LustreError: 391989:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1548288(2596864)  req@ffff8ed717ad2850 x1654553683462544/t0(0) o4->2675d9e5-c3be-60e2-3a72-ea55143ec798@10.210.12.25@tcp1:238/0 lens 488/448 e 0 to 0 dl 1603130998 ref 1 fl Interpret:/2/0 rc 0/0
Oct 19 11:09:26 oak-io5-s2 kernel: LustreError: 391989:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
Oct 19 11:09:41 oak-io5-s2 kernel: LustreError: 347892:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff8ecdf37ee850 x1678163504405760/t0(0) o4->a8bf7f45-a2e0-4@10.50.15.5@o2ib2:249/0 lens 488/448 e 0 to 0 dl 1603131009 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 11:09:41 oak-io5-s2 kernel: LustreError: 347892:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
Oct 19 11:09:46 oak-io5-s2 kernel: LustreError: 454788:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff8e336449b850 x1674949592590496/t0(0) o3->6c0c453f-2604-6818-784d-e05fcb7efca9@10.210.12.129@tcp1:258/0 lens 488/440 e 0 to 0 dl 1603131018 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 11:09:46 oak-io5-s2 kernel: Lustre: oak-OST0108: Bulk IO read error with f73489af-edd7-0720-ca37-908500000b7c (at 10.210.12.131@tcp1), client will retry: rc -110
Oct 19 11:09:46 oak-io5-s2 kernel: Lustre: Skipped 19 previous similar messages
Oct 19 11:09:46 oak-io5-s2 kernel: LustreError: 454788:0:(ldlm_lib.c:3294:target_bulk_io()) Skipped 5 previous similar messages
Oct 19 11:10:01 oak-io5-s2 kernel: LustreError: 347874:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff8e336449f850 x1669823930986624/t0(0) o4->bf7f3100-aa36-e17d-473f-b840b304ff3e@10.210.12.31@tcp1:273/0 lens 488/448 e 0 to 0 dl 1603131033 ref 1 fl Interpret:/2/0 rc 0/0
Oct 19 11:10:01 oak-io5-s2 kernel: LustreError: 347808:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff8e7597e61050 x1669823930986784/t0(0) o4->bf7f3100-aa36-e17d-473f-b840b304ff3e@10.210.12.31@tcp1:273/0 lens 488/448 e 0 to 0 dl 1603131033 ref 1 fl Interpret:/2/0 rc 0/0
Oct 19 11:10:01 oak-io5-s2 kernel: LustreError: 454788:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff8e2ee63b3850 x1669688660365008/t0(0) o3->f73489af-edd7-0720-ca37-908500000b7c@10.210.12.131@tcp1:270/0 lens 488/440 e 0 to 0 dl 1603131030 ref 1 fl Interpret:/2/0 rc 0/0
Oct 19 11:10:01 oak-io5-s2 kernel: LustreError: 454788:0:(ldlm_lib.c:3294:target_bulk_io()) Skipped 1 previous similar message
Oct 19 11:10:01 oak-io5-s2 kernel: Lustre: oak-OST0108: Bulk IO read error with f73489af-edd7-0720-ca37-908500000b7c (at 10.210.12.131@tcp1), client will retry: rc -110
Oct 19 11:10:01 oak-io5-s2 kernel: Lustre: Skipped 5 previous similar messages
Oct 19 11:10:01 oak-io5-s2 kernel: LustreError: 347874:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
Oct 19 11:10:02 oak-io5-s2 kernel: LustreError: 347881:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8e75a6464050 x1677892668536224/t0(0) o4->548d000f-ab1a-9806-8901-4ccc0d3ca2ee@10.210.12.19@tcp1:281/0 lens 488/448 e 0 to 0 dl 1603131041 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 11:10:02 oak-io5-s2 kernel: LustreError: 347881:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 28 previous similar messages
Oct 19 11:11:21 oak-io5-s2 kernel: LustreError: 94042:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff8ecdfdbb0850 x1678105846220800/t0(0) o4->9c2514d9-52ce-4@10.50.12.13@o2ib2:354/0 lens 488/448 e 0 to 0 dl 1603131114 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 11:11:21 oak-io5-s2 kernel: LustreError: 94042:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
Oct 19 11:11:21 oak-io5-s2 kernel: Lustre: oak-OST010b: Bulk IO write error with 9c2514d9-52ce-4 (at 10.50.12.13@o2ib2), client will retry: rc = -110
Oct 19 11:11:21 oak-io5-s2 kernel: Lustre: Skipped 50 previous similar messages
Oct 19 11:11:50 oak-io5-s2 kernel: Lustre: oak-OST0108: haven't heard from client 97cee6fe-a419-4 (at 10.49.27.5@o2ib1) in 215 seconds. I think it's dead, and I am evicting it. exp ffff8e740f349000, cur 1603131110 expire 1603130960 last 1603130895
Oct 19 11:11:50 oak-io5-s2 kernel: Lustre: Skipped 17150 previous similar messages
Oct 19 11:18:56 oak-io5-s2 kernel: Lustre: oak-OST00c5: Connection restored to 6caa2f40-eaae-4 (at 10.51.1.72@o2ib3)
Oct 19 11:18:56 oak-io5-s2 kernel: Lustre: oak-OST00c3: Connection restored to 6caa2f40-eaae-4 (at 10.51.1.72@o2ib3)
Oct 19 11:18:56 oak-io5-s2 kernel: Lustre: oak-OST00c1: Connection restored to 6caa2f40-eaae-4 (at 10.51.1.72@o2ib3)
Oct 19 11:18:56 oak-io5-s2 kernel: Lustre: Skipped 84635 previous similar messages
Oct 19 11:18:56 oak-io5-s2 kernel: Lustre: Skipped 84635 previous similar messages
Oct 19 11:18:56 oak-io5-s2 kernel: Lustre: Skipped 35 previous similar messages
Oct 19 11:20:37 oak-io5-s2 kernel: Lustre: oak-OST00eb: Client 2675d9e5-c3be-60e2-3a72-ea55143ec798 (at 10.210.12.25@tcp1) reconnecting
Oct 19 11:20:37 oak-io5-s2 kernel: Lustre: Skipped 79257 previous similar messages
Oct 19 11:23:51 oak-io5-s2 kernel: perf: interrupt took too long (2512 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
Oct 19 11:26:06 oak-io5-s2 kernel: Lustre: oak-OST00c5: haven't heard from client 5e2389f5-4ddb-4 (at 10.49.26.26@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6cceb55800, cur 1603131966 expire 1603131816 last 1603131739
Oct 19 11:26:06 oak-io5-s2 kernel: Lustre: Skipped 5110 previous similar messages
Oct 19 11:30:16 oak-io5-s2 kernel: Lustre: oak-OST00c1: Connection restored to  (at 10.49.26.26@o2ib1)
Oct 19 11:30:16 oak-io5-s2 kernel: Lustre: Skipped 165 previous similar messages
Oct 19 11:50:42 oak-io5-s2 kernel: Lustre: oak-OST00e4: haven't heard from client b059364d-78b1-4 (at 10.50.15.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6ed9fb4400, cur 1603133442 expire 1603133292 last 1603133215
Oct 19 11:50:42 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 11:50:49 oak-io5-s2 kernel: perf: interrupt took too long (3142 > 3140), lowering kernel.perf_event_max_sample_rate to 63000
Oct 19 11:57:56 oak-io5-s2 kernel: Lustre: oak-OST0102: haven't heard from client 0c171b31-32d0-4 (at 10.50.0.63@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ed7300df400, cur 1603133876 expire 1603133726 last 1603133649
Oct 19 11:57:56 oak-io5-s2 kernel: Lustre: Skipped 159 previous similar messages
Oct 19 12:09:58 oak-io5-s2 kernel: LustreError: 347800:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00d2: cli 26503748-f849-4 claims 4218880 GRANT, real grant 4194304
Oct 19 12:09:58 oak-io5-s2 kernel: LustreError: 347800:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 1 previous similar message
Oct 19 12:10:23 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 12:10:23 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603134323, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e865484de80/0xf1ba1949935decd lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ce9950356b expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 12:10:23 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 12:10:23 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 12:10:23 oak-io5-s2 kernel: Lustre: Skipped 40 previous similar messages
Oct 19 12:12:22 oak-io5-s2 kernel: Lustre: oak-OST00f4: haven't heard from client 89f5420c-a7ea-4 (at 10.49.7.15@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e760a7fd400, cur 1603134742 expire 1603134592 last 1603134515
Oct 19 12:12:22 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 12:12:50 oak-io5-s2 kernel: Lustre: oak-OST00fd: haven't heard from client 89f5420c-a7ea-4 (at 10.49.7.15@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7a706fec00, cur 1603134770 expire 1603134620 last 1603134543
Oct 19 12:12:50 oak-io5-s2 kernel: Lustre: Skipped 467 previous similar messages
Oct 19 12:15:23 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 12:15:23 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603134623, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8eaec946f080/0xf1ba194993e392a lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ce998b53cd expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 12:15:23 oak-io5-s2 kernel: LustreError: 128423:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e637518ed80) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 12:15:23 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 12:15:23 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 12:20:23 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 12:20:23 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603134923, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8ea131c021c0/0xf1ba19499408394 lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ce99d3f4e5 expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 12:20:23 oak-io5-s2 kernel: LustreError: 177061:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8ea8f7203740) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 12:20:23 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 12:20:23 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 12:24:07 oak-io5-s2 kernel: Lustre: oak-OST00d2: haven't heard from client 8bf9aaae-3a35-4 (at 10.50.6.53@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ed9563ed400, cur 1603135447 expire 1603135297 last 1603135220
Oct 19 12:24:07 oak-io5-s2 kernel: Lustre: Skipped 11 previous similar messages
Oct 19 12:24:15 oak-io5-s2 kernel: Lustre: oak-OST00ec: haven't heard from client 8bf9aaae-3a35-4 (at 10.50.6.53@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8ecbaf36a800, cur 1603135455 expire 1603135305 last 1603135228
Oct 19 12:25:24 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 12:25:24 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603135223, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6cbf308480/0xf1ba1949943004e lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ce9a268477 expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 12:25:24 oak-io5-s2 kernel: LustreError: 225988:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e2ee7e20780) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 12:25:24 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 12:25:24 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 12:30:24 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 12:30:24 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603135524, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e864107fbc0/0xf1ba194994e739c lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ce9a7dbc01 expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 12:30:24 oak-io5-s2 kernel: LustreError: 275339:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8ea7ce108cc0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 12:30:24 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 12:35:24 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 12:35:24 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603135824, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e2505962640/0xf1ba19499569a8a lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ce9ac12841 expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 12:35:24 oak-io5-s2 kernel: LustreError: 325103:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e3218923080) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 12:35:24 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 12:35:24 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 12:35:24 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 12:40:24 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 12:40:24 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603136124, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8ea03dae3600/0xf1ba194995f9d81 lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ce9b087b2e expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 12:40:24 oak-io5-s2 kernel: LustreError: 375509:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e9c4f6743c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 12:40:24 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 12:45:24 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 12:45:24 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603136424, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e9fee46c5c0/0xf1ba1949967ae81 lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ce9b64d496 expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 12:45:24 oak-io5-s2 kernel: LustreError: 426439:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e99f74832c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 12:45:24 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 12:45:24 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 12:45:24 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 12:48:29 oak-io5-s2 kernel: perf: interrupt took too long (3934 > 3927), lowering kernel.perf_event_max_sample_rate to 50000
Oct 19 12:50:02 oak-io5-s2 kernel: Lustre: oak-OST00ce: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e78365ec000, cur 1603137002 expire 1603136852 last 1603136775
Oct 19 12:50:02 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 19 12:50:24 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 12:50:24 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603136724, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e3e78bc06c0/0xf1ba194996fc6b2 lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ce9bc247da expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 12:50:24 oak-io5-s2 kernel: LustreError: 18822:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e5e61c13bc0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 12:50:24 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 12:51:32 oak-io5-s2 kernel: Lustre: oak-OST00d3: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e741395ec00, cur 1603137092 expire 1603136942 last 1603136865
Oct 19 12:54:00 oak-io5-s2 kernel: Lustre: oak-OST00db: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6d4ab32800, cur 1603137240 expire 1603137090 last 1603137013
Oct 19 12:54:00 oak-io5-s2 kernel: Lustre: Skipped 65 previous similar messages
Oct 19 12:54:12 oak-io5-s2 kernel: Lustre: oak-OST00ce: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7a44cbc800, cur 1603137252 expire 1603137102 last 1603137025
Oct 19 12:54:12 oak-io5-s2 kernel: Lustre: Skipped 11 previous similar messages
Oct 19 12:54:35 oak-io5-s2 kernel: Lustre: oak-OST0105: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6df476ac00, cur 1603137275 expire 1603137125 last 1603137048
Oct 19 12:55:24 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 12:55:24 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603137024, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e9f95478000/0xf1ba194997be87b lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ce9c489b53 expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 12:55:24 oak-io5-s2 kernel: LustreError: 73264:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e430a23eb40) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 12:55:24 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 12:55:24 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 12:55:24 oak-io5-s2 kernel: Lustre: Skipped 2 previous similar messages
Oct 19 12:59:11 oak-io5-s2 kernel: mpt3sas_cm1: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
Oct 19 13:00:24 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 13:00:24 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603137324, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8ec1081dee40/0xf1ba194998808b5 lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ce9cec1081 expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 13:00:24 oak-io5-s2 kernel: LustreError: 124407:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e8f4745bc80) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 13:00:24 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 13:05:24 oak-io5-s2 kernel: LustreError: 175892:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e31cd7dbec0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 13:05:24 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 13:05:24 oak-io5-s2 kernel: Lustre: Skipped 35 previous similar messages
Oct 19 13:07:02 oak-io5-s2 kernel: Lustre: oak-OST00e9: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7a1e623800, cur 1603138022 expire 1603137872 last 1603137795
Oct 19 13:07:13 oak-io5-s2 kernel: Lustre: oak-OST00c2: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6ccaed1400, cur 1603138033 expire 1603137883 last 1603137806
Oct 19 13:07:13 oak-io5-s2 kernel: Lustre: Skipped 9 previous similar messages
Oct 19 13:07:19 oak-io5-s2 kernel: Lustre: oak-OST00d0: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7a3fa40800, cur 1603138039 expire 1603137889 last 1603137812
Oct 19 13:07:19 oak-io5-s2 kernel: Lustre: Skipped 3 previous similar messages
Oct 19 13:08:04 oak-io5-s2 kernel: Lustre: oak-OST00c6: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6cd5292400, cur 1603138084 expire 1603137934 last 1603137857
Oct 19 13:08:04 oak-io5-s2 kernel: Lustre: Skipped 3 previous similar messages
Oct 19 13:08:31 oak-io5-s2 kernel: Lustre: oak-OST00cd: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e705d6b0000, cur 1603138111 expire 1603137961 last 1603137884
Oct 19 13:08:31 oak-io5-s2 kernel: Lustre: Skipped 6 previous similar messages
Oct 19 13:08:41 oak-io5-s2 kernel: LustreError: 347818:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST010b: cli c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 claims 4218880 GRANT, real grant 1314816
Oct 19 13:08:58 oak-io5-s2 kernel: LustreError: 347797:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST010b: cli c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 claims 1728512 GRANT, real grant 0
Oct 19 13:08:58 oak-io5-s2 kernel: LustreError: 347797:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 4 previous similar messages
Oct 19 13:09:15 oak-io5-s2 kernel: Lustre: oak-OST00fa: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6cc57c6c00, cur 1603138155 expire 1603138005 last 1603137928
Oct 19 13:09:15 oak-io5-s2 kernel: Lustre: Skipped 8 previous similar messages
Oct 19 13:10:24 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 13:10:24 oak-io5-s2 kernel: LustreError: Skipped 1 previous similar message
Oct 19 13:10:24 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603137924, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e38ec4433c0/0xf1ba19499a44618 lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ce9e4b0aa5 expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 13:10:24 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Oct 19 13:10:24 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 13:10:24 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) Skipped 1 previous similar message
Oct 19 13:10:50 oak-io5-s2 kernel: Lustre: oak-OST00fe: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7419fa6800, cur 1603138250 expire 1603138100 last 1603138023
Oct 19 13:10:50 oak-io5-s2 kernel: Lustre: Skipped 36 previous similar messages
Oct 19 13:11:56 oak-io5-s2 kernel: LustreError: 73007:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST010b: cli c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 claims 2428928 GRANT, real grant 0
Oct 19 13:11:56 oak-io5-s2 kernel: LustreError: 73007:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 3 previous similar messages
Oct 19 13:12:11 oak-io5-s2 kernel: LustreError: 453754:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST010b: cli c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 claims 2547712 GRANT, real grant 0
Oct 19 13:12:21 oak-io5-s2 kernel: LustreError: 107340:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST010b: cli c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 claims 1748992 GRANT, real grant 0
Oct 19 13:12:36 oak-io5-s2 kernel: LustreError: 64884:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST010b: cli c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 claims 1847296 GRANT, real grant 0
Oct 19 13:12:36 oak-io5-s2 kernel: LustreError: 64884:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 1 previous similar message
Oct 19 13:12:57 oak-io5-s2 kernel: LustreError: 63789:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST010b: cli c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 claims 868352 GRANT, real grant 0
Oct 19 13:12:57 oak-io5-s2 kernel: LustreError: 63789:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 2 previous similar messages
Oct 19 13:14:35 oak-io5-s2 kernel: Lustre: oak-OST0105: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e62e0b8c400, cur 1603138475 expire 1603138325 last 1603138248
Oct 19 13:14:35 oak-io5-s2 kernel: Lustre: Skipped 7 previous similar messages
Oct 19 13:15:24 oak-io5-s2 kernel: LustreError: 232627:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e4fb2369b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 13:15:24 oak-io5-s2 kernel: LustreError: 232627:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message
Oct 19 13:15:24 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 13:15:24 oak-io5-s2 kernel: Lustre: Skipped 47 previous similar messages
Oct 19 13:19:15 oak-io5-s2 kernel: mpt3sas_cm1: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303)
Oct 19 13:20:24 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 13:20:24 oak-io5-s2 kernel: LustreError: Skipped 1 previous similar message
Oct 19 13:20:24 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603138524, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e33b03f0d80/0xf1ba19499c0195c lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ce9fca905f expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 13:20:24 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Oct 19 13:20:24 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 13:20:24 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) Skipped 1 previous similar message
Oct 19 13:20:28 oak-io5-s2 kernel: LustreError: 65289:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST010b: cli c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 claims 4218880 GRANT, real grant 0
Oct 19 13:20:28 oak-io5-s2 kernel: LustreError: 65289:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 3 previous similar messages
Oct 19 13:23:24 oak-io5-s2 kernel: LustreError: 347794:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST010b: cli c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 claims 4218880 GRANT, real grant 0
Oct 19 13:23:24 oak-io5-s2 kernel: LustreError: 347794:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 2 previous similar messages
Oct 19 13:25:24 oak-io5-s2 kernel: LustreError: 281099:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e9b29731b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 13:25:24 oak-io5-s2 kernel: LustreError: 281099:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message
Oct 19 13:25:24 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 13:25:24 oak-io5-s2 kernel: Lustre: Skipped 34 previous similar messages
Oct 19 13:27:03 oak-io5-s2 kernel: Lustre: oak-OST00ed: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6ccefc3400, cur 1603139223 expire 1603139073 last 1603138996
Oct 19 13:30:16 oak-io5-s2 kernel: Lustre: 418340:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1603139409/real 1603139409]  req@ffff8e6cd7013600 x1680988720228224/t0(0) o104->oak-OST00f7@10.210.9.195@tcp1:15/16 lens 296/224 e 0 to 1 dl 1603139416 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Oct 19 13:30:16 oak-io5-s2 kernel: Lustre: 418340:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 353 previous similar messages
Oct 19 13:30:21 oak-io5-s2 kernel: Lustre: oak-OST00f7: Client c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1) reconnecting
Oct 19 13:30:21 oak-io5-s2 kernel: LustreError: 347845:0:(ldlm_lib.c:3228:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff8e40582d1850 x1678199497425440/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:404/0 lens 488/448 e 0 to 0 dl 1603139469 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 13:30:21 oak-io5-s2 kernel: Lustre: oak-OST00e4: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -107
Oct 19 13:30:21 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 13:30:21 oak-io5-s2 kernel: Lustre: Skipped 2 previous similar messages
Oct 19 13:30:21 oak-io5-s2 kernel: LustreError: 174513:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8e705754f850 x1678199497189792/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:376/0 lens 488/448 e 1 to 0 dl 1603139441 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 13:30:21 oak-io5-s2 kernel: LustreError: 174513:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 6 previous similar messages
Oct 19 13:30:24 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 13:30:24 oak-io5-s2 kernel: LustreError: Skipped 1 previous similar message
Oct 19 13:30:24 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603139124, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8ebed938b840/0xf1ba19499db4ec5 lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0cea17304fd expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 13:30:24 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Oct 19 13:30:24 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 13:30:24 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) Skipped 1 previous similar message
Oct 19 13:35:08 oak-io5-s2 kernel: LustreError: 107338:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST010b: cli c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 claims 45056 GRANT, real grant 0
Oct 19 13:35:08 oak-io5-s2 kernel: LustreError: 107338:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 17 previous similar messages
Oct 19 13:35:24 oak-io5-s2 kernel: LustreError: 329382:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e3519cf1380) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 13:35:24 oak-io5-s2 kernel: LustreError: 329382:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message
Oct 19 13:35:24 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 13:35:25 oak-io5-s2 kernel: Lustre: Skipped 54 previous similar messages
Oct 19 13:40:25 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 13:40:25 oak-io5-s2 kernel: LustreError: Skipped 1 previous similar message
Oct 19 13:40:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603139724, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8ecdd2b7c140/0xf1ba19499ff99c4 lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0cea371828f expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 13:40:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Oct 19 13:40:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 13:40:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) Skipped 1 previous similar message
Oct 19 13:40:38 oak-io5-s2 kernel: LustreError: 64539:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST010b: cli c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 claims 2920448 GRANT, real grant 0
Oct 19 13:40:38 oak-io5-s2 kernel: LustreError: 64539:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 8 previous similar messages
Oct 19 13:45:25 oak-io5-s2 kernel: LustreError: 377816:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8ea1a0360900) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 13:45:25 oak-io5-s2 kernel: LustreError: 377816:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message
Oct 19 13:45:25 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 13:45:25 oak-io5-s2 kernel: Lustre: Skipped 35 previous similar messages
Oct 19 13:47:03 oak-io5-s2 kernel: Lustre: oak-OST00ed: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6cd88f1c00, cur 1603140423 expire 1603140273 last 1603140196
Oct 19 13:47:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 13:50:25 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 13:50:25 oak-io5-s2 kernel: LustreError: Skipped 1 previous similar message
Oct 19 13:50:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603140325, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e3a19615100/0xf1ba1949a1c231d lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0cea59e70dd expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 13:50:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Oct 19 13:50:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 13:50:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) Skipped 1 previous similar message
Oct 19 13:53:28 oak-io5-s2 kernel: LustreError: 347797:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST010b: cli c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 claims 4218880 GRANT, real grant 0
Oct 19 13:53:28 oak-io5-s2 kernel: LustreError: 347797:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 6 previous similar messages
Oct 19 13:55:25 oak-io5-s2 kernel: LustreError: 426713:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e3ba9ab89c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 13:55:25 oak-io5-s2 kernel: LustreError: 426713:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message
Oct 19 13:55:25 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 13:55:25 oak-io5-s2 kernel: Lustre: Skipped 47 previous similar messages
Oct 19 13:58:40 oak-io5-s2 kernel: Lustre: 417068:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1603141113/real 1603141113]  req@ffff8ec7c6c81f80 x1680988770240192/t0(0) o104->oak-OST0106@10.210.9.195@tcp1:15/16 lens 296/224 e 0 to 1 dl 1603141120 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Oct 19 14:00:25 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 14:00:25 oak-io5-s2 kernel: LustreError: Skipped 1 previous similar message
Oct 19 14:00:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603140925, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4d96c3d580/0xf1ba1949a41a628 lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0cea801dc9c expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 14:00:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Oct 19 14:00:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 14:00:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) Skipped 1 previous similar message
Oct 19 14:05:25 oak-io5-s2 kernel: LustreError: 16861:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e8deddad2c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 14:05:25 oak-io5-s2 kernel: LustreError: 16861:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message
Oct 19 14:05:25 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 14:05:25 oak-io5-s2 kernel: Lustre: Skipped 35 previous similar messages
Oct 19 14:07:03 oak-io5-s2 kernel: Lustre: oak-OST00ed: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6ce5d2d400, cur 1603141623 expire 1603141473 last 1603141396
Oct 19 14:07:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 14:10:25 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 14:10:25 oak-io5-s2 kernel: LustreError: Skipped 1 previous similar message
Oct 19 14:10:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603141525, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e55150f4a40/0xf1ba1949a55c51e lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0cea91a5a4f expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 14:10:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Oct 19 14:10:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 14:10:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) Skipped 1 previous similar message
Oct 19 14:12:05 oak-io5-s2 kernel: Lustre: oak-OST00f7: Client c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1) reconnecting
Oct 19 14:12:05 oak-io5-s2 kernel: Lustre: Skipped 4 previous similar messages
Oct 19 14:12:06 oak-io5-s2 kernel: LustreError: 347807:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8e253b62d850 x1678199512154080/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:625/0 lens 488/448 e 1 to 0 dl 1603141955 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 14:12:06 oak-io5-s2 kernel: Lustre: oak-OST00f7: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 14:12:06 oak-io5-s2 kernel: Lustre: Skipped 10 previous similar messages
Oct 19 14:12:06 oak-io5-s2 kernel: LustreError: 347807:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 10 previous similar messages
Oct 19 14:15:25 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 14:15:25 oak-io5-s2 kernel: Lustre: Skipped 49 previous similar messages
Oct 19 14:20:25 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 14:20:25 oak-io5-s2 kernel: LustreError: Skipped 1 previous similar message
Oct 19 14:20:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603142125, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e2ffde94800/0xf1ba1949a741856 lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ceaa190ce0 expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 14:20:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Oct 19 14:20:25 oak-io5-s2 kernel: LustreError: 93053:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e259a7bc3c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 14:20:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 14:20:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) Skipped 1 previous similar message
Oct 19 14:22:42 oak-io5-s2 kernel: Lustre: oak-OST00eb: haven't heard from client 529b0362-bb36-52d3-6dc0-bc9c3d9dafc4 (at 10.210.12.86@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e62e8531000, cur 1603142562 expire 1603142412 last 1603142335
Oct 19 14:22:42 oak-io5-s2 kernel: Lustre: Skipped 159 previous similar messages
Oct 19 14:25:25 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 14:25:25 oak-io5-s2 kernel: Lustre: Skipped 35 previous similar messages
Oct 19 14:30:25 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 14:30:25 oak-io5-s2 kernel: LustreError: Skipped 1 previous similar message
Oct 19 14:30:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603142725, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e412983da00/0xf1ba1949a8f0bc4 lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ceab01a05c expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 14:30:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Oct 19 14:30:25 oak-io5-s2 kernel: LustreError: 141608:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e446398c300) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 14:30:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 14:30:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) Skipped 1 previous similar message
Oct 19 14:30:25 oak-io5-s2 kernel: LustreError: 141608:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message
Oct 19 14:34:34 oak-io5-s2 kernel: Lustre: oak-OST0105: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e76167ac000, cur 1603143274 expire 1603143124 last 1603143047
Oct 19 14:34:34 oak-io5-s2 kernel: Lustre: Skipped 158 previous similar messages
Oct 19 14:35:25 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 14:35:25 oak-io5-s2 kernel: Lustre: Skipped 47 previous similar messages
Oct 19 14:40:25 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 14:40:25 oak-io5-s2 kernel: LustreError: Skipped 1 previous similar message
Oct 19 14:40:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603143325, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8eb425acc800/0xf1ba1949aaba0fb lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ceabe37586 expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 14:40:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Oct 19 14:40:25 oak-io5-s2 kernel: LustreError: 190442:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e975d155e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 14:40:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 14:40:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) Skipped 1 previous similar message
Oct 19 14:40:25 oak-io5-s2 kernel: LustreError: 190442:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message
Oct 19 14:45:25 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 14:45:25 oak-io5-s2 kernel: Lustre: Skipped 35 previous similar messages
Oct 19 14:47:02 oak-io5-s2 kernel: Lustre: oak-OST00ed: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6cc20d7800, cur 1603144022 expire 1603143872 last 1603143795
Oct 19 14:50:25 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 14:50:25 oak-io5-s2 kernel: LustreError: Skipped 1 previous similar message
Oct 19 14:50:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603143925, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e3f3e1cd7c0/0xf1ba1949ac840ab lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ceace03a19 expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 14:50:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Oct 19 14:50:25 oak-io5-s2 kernel: LustreError: 238874:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e30042df380) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 14:50:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 14:50:25 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) Skipped 1 previous similar message
Oct 19 14:50:25 oak-io5-s2 kernel: LustreError: 238874:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message
Oct 19 14:55:25 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 14:55:25 oak-io5-s2 kernel: Lustre: Skipped 47 previous similar messages
Oct 19 15:00:25 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 15:00:25 oak-io5-s2 kernel: LustreError: Skipped 1 previous similar message
Oct 19 15:00:25 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603144525, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e3694e2d100/0xf1ba1949af3c6dd lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ceade81f4a expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 15:00:26 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Oct 19 15:00:26 oak-io5-s2 kernel: LustreError: 287342:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8ebcc379c3c0) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 15:00:26 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 15:00:26 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) Skipped 1 previous similar message
Oct 19 15:00:26 oak-io5-s2 kernel: LustreError: 287342:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message
Oct 19 15:05:26 oak-io5-s2 kernel: Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5)
Oct 19 15:05:26 oak-io5-s2 kernel: Lustre: Skipped 35 previous similar messages
Oct 19 15:07:03 oak-io5-s2 kernel: Lustre: oak-OST010f: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6d4aa0bc00, cur 1603145223 expire 1603145073 last 1603144996
Oct 19 15:07:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 15:07:23 oak-io5-s2 kernel: perf: interrupt took too long (4918 > 4917), lowering kernel.perf_event_max_sample_rate to 40000
Oct 19 15:10:26 oak-io5-s2 kernel: LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
Oct 19 15:10:26 oak-io5-s2 kernel: LustreError: Skipped 1 previous similar message
Oct 19 15:10:26 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1603145126, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e951405e300/0xf1ba1949b184926 lrc: 4/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xe88ce0ceae93def2 expref: -99 pid: 416739 timeout: 0 lvb_type: 0
Oct 19 15:10:26 oak-io5-s2 kernel: LustreError: 416739:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Oct 19 15:10:26 oak-io5-s2 kernel: LustreError: 335844:0:(ldlm_resource.c:1147:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x736d61726170:0x3:0x0].0x0 (ffff8e46d4fe0240) refcount nonzero (1) after lock cleanup; forcing cleanup.
Oct 19 15:10:26 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) failed processing log: -5
Oct 19 15:10:26 oak-io5-s2 kernel: LustreError: 416739:0:(mgc_request.c:599:do_requeue()) Skipped 1 previous similar message
Oct 19 15:10:26 oak-io5-s2 kernel: LustreError: 335844:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 1 previous similar message
Oct 19 15:12:59 oak-io5-s2 kernel: Lustre: oak-OST00c9: Client 7abf3cb3-63e9-4 (at 10.50.1.24@o2ib2) reconnecting
Oct 19 15:12:59 oak-io5-s2 kernel: Lustre: Skipped 2 previous similar messages
Oct 19 15:13:00 oak-io5-s2 kernel: Lustre: 28614:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1603145576/real 1603145579]  req@ffff8ecded73d580 x1680988932768192/t0(0) o400->oak-MDT0001-lwp-OST0104@10.0.2.51@o2ib5:12/10 lens 224/224 e 0 to 1 dl 1603145832 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1
Oct 19 15:13:00 oak-io5-s2 kernel: Lustre: oak-MDT0002-lwp-OST010e: Connection to oak-MDT0002 (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
Oct 19 15:13:00 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 15:13:00 oak-io5-s2 kernel: Lustre: 28614:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Oct 19 15:13:05 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds
Oct 19 15:13:05 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Skipped 1 previous similar message
Oct 19 15:13:05 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.2.51@o2ib5 (6): c: 0, oc: 0, rc: 8
Oct 19 15:13:05 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Skipped 1 previous similar message
Oct 19 15:13:11 oak-io5-s2 kernel: Lustre: 28614:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1603145576/real 1603145591]  req@ffff8e7afc20f980 x1680988932771264/t0(0) o400->oak-MDT0001-lwp-OST00e0@10.0.2.51@o2ib5:12/10 lens 224/224 e 0 to 1 dl 1603145832 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1
Oct 19 15:13:11 oak-io5-s2 kernel: Lustre: oak-MDT0002-lwp-OST00f8: Connection to oak-MDT0002 (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
Oct 19 15:13:11 oak-io5-s2 kernel: Lustre: Skipped 5 previous similar messages
Oct 19 15:13:11 oak-io5-s2 kernel: Lustre: 28614:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Oct 19 15:13:17 oak-io5-s2 kernel: Lustre: 28614:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1603145576/real 1603145597]  req@ffff8e5cc4933a80 x1680988932772800/t0(0) o400->oak-MDT0001-lwp-OST00d4@10.0.2.51@o2ib5:12/10 lens 224/224 e 0 to 1 dl 1603145832 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1
Oct 19 15:13:17 oak-io5-s2 kernel: Lustre: oak-MDT0002-lwp-OST00c8: Connection to oak-MDT0002 (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
Oct 19 15:13:17 oak-io5-s2 kernel: Lustre: Skipped 6 previous similar messages
Oct 19 15:13:17 oak-io5-s2 kernel: Lustre: 28614:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Oct 19 15:13:23 oak-io5-s2 kernel: Lustre: 28614:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1603145576/real 1603145603]  req@ffff8e7b0179f500 x1680988932774336/t0(0) o400->oak-MDT0001-lwp-OST00c2@10.0.2.51@o2ib5:12/10 lens 224/224 e 0 to 1 dl 1603145832 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1
Oct 19 15:13:23 oak-io5-s2 kernel: Lustre: oak-MDT0002-lwp-OST00e2: Connection to oak-MDT0002 (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
Oct 19 15:13:23 oak-io5-s2 kernel: Lustre: Skipped 8 previous similar messages
Oct 19 15:13:23 oak-io5-s2 kernel: Lustre: 28614:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Oct 19 15:13:29 oak-io5-s2 kernel: Lustre: 28614:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1603145576/real 1603145609]  req@ffff8e4b852bbf00 x1680988932775872/t0(0) o400->oak-MDT0001-lwp-OST00d5@10.0.2.51@o2ib5:12/10 lens 224/224 e 0 to 1 dl 1603145832 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1
Oct 19 15:13:29 oak-io5-s2 kernel: Lustre: 28614:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Oct 19 15:13:35 oak-io5-s2 kernel: Lustre: oak-MDT0001-lwp-OST00ef: Connection to oak-MDT0001 (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
Oct 19 15:13:35 oak-io5-s2 kernel: Lustre: Skipped 21 previous similar messages
Oct 19 15:13:36 oak-io5-s2 kernel: LNet: 28355:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.2.51@o2ib5: 1 seconds
Oct 19 15:13:36 oak-io5-s2 kernel: LNet: 28355:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 5 previous similar messages
Oct 19 15:13:39 oak-io5-s2 kernel: Lustre: oak-OST00c1: Client a8c54898-612c-e786-e8c4-7100b7acf565 (at 10.210.15.2@tcp1) reconnecting
Oct 19 15:13:39 oak-io5-s2 kernel: Lustre: Skipped 50 previous similar messages
Oct 19 15:13:39 oak-io5-s2 kernel: Lustre: 28618:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1603145576/real 1603145619]  req@ffff8e7aecbbf500 x1680988932779200/t0(0) o400->oak-MDT0002-lwp-OST00e9@10.0.2.51@o2ib5:12/10 lens 224/224 e 0 to 1 dl 1603145832 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1
Oct 19 15:13:39 oak-io5-s2 kernel: Lustre: 28618:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 21 previous similar messages
Oct 19 15:13:46 oak-io5-s2 kernel: LNet: 28355:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.2.51@o2ib5: 11 seconds
Oct 19 15:13:46 oak-io5-s2 kernel: LNet: 28355:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 41 previous similar messages
Oct 19 15:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c2: Client a8c54898-612c-e786-e8c4-7100b7acf565 (at 10.210.15.2@tcp1) reconnecting
Oct 19 15:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c0: Client a8c54898-612c-e786-e8c4-7100b7acf565 (at 10.210.15.2@tcp1) reconnecting
Oct 19 15:13:50 oak-io5-s2 kernel: Lustre: Skipped 46 previous similar messages
Oct 19 15:13:51 oak-io5-s2 kernel: Lustre: oak-MDT0002-lwp-OST00cc: Connection to oak-MDT0002 (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
Oct 19 15:13:51 oak-io5-s2 kernel: Lustre: Skipped 63 previous similar messages
Oct 19 15:13:56 oak-io5-s2 kernel: Lustre: 28607:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1603145597/real 1603145636]  req@ffff8e7b38741b00 x1680988933275456/t0(0) o400->oak-MDT0002-lwp-OST00f2@10.0.2.51@o2ib5:12/10 lens 224/224 e 0 to 1 dl 1603145853 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1
Oct 19 15:13:56 oak-io5-s2 kernel: Lustre: 28607:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 74 previous similar messages
Oct 19 15:14:00 oak-io5-s2 kernel: LustreError: 347838:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8ed7144d7850 x1680106673358208/t0(0) o4->4fd07171-e2d6-5bc3-8cbf-b8edd4ce0c59@10.210.12.9@tcp1:580/0 lens 488/448 e 0 to 0 dl 1603145685 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:14:00 oak-io5-s2 kernel: Lustre: oak-OST00e8: Bulk IO write error with 4fd07171-e2d6-5bc3-8cbf-b8edd4ce0c59 (at 10.210.12.9@tcp1), client will retry: rc = -110
Oct 19 15:14:00 oak-io5-s2 kernel: LustreError: 62346:0:(ldlm_lib.c:3228:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff8ece2f7b4850 x1680106673358208/t0(0) o4->4fd07171-e2d6-5bc3-8cbf-b8edd4ce0c59@10.210.12.9@tcp1:584/0 lens 488/448 e 0 to 0 dl 1603145689 ref 1 fl Interpret:/2/0 rc 0/0
Oct 19 15:14:00 oak-io5-s2 kernel: Lustre: Skipped 2 previous similar messages
Oct 19 15:14:00 oak-io5-s2 kernel: Lustre: oak-OST00dd: Client 4fd07171-e2d6-5bc3-8cbf-b8edd4ce0c59 (at 10.210.12.9@tcp1) reconnecting
Oct 19 15:14:00 oak-io5-s2 kernel: Lustre: Skipped 187 previous similar messages
Oct 19 15:14:05 oak-io5-s2 kernel: LNet: 28355:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.2.51@o2ib5: 18 seconds
Oct 19 15:14:05 oak-io5-s2 kernel: LNet: 28355:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 96 previous similar messages
Oct 19 15:14:19 oak-io5-s2 kernel: LustreError: 347882:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8ebdb843b850 x1678216579519872/t0(0) o4->181aa8b0-7375-4@10.50.13.10@o2ib2:603/0 lens 488/448 e 0 to 0 dl 1603145708 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:14:19 oak-io5-s2 kernel: LustreError: 347882:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 1 previous similar message
Oct 19 15:14:19 oak-io5-s2 kernel: Lustre: oak-OST00d2: Bulk IO write error with 181aa8b0-7375-4 (at 10.50.13.10@o2ib2), client will retry: rc = -110
Oct 19 15:14:19 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 15:14:19 oak-io5-s2 kernel: Lustre: oak-OST00d5: Client 791a5387-8d55-4 (at 10.49.25.29@o2ib1) reconnecting
Oct 19 15:14:19 oak-io5-s2 kernel: Lustre: Skipped 294 previous similar messages
Oct 19 15:14:22 oak-io5-s2 kernel: LustreError: 73006:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 3145728(4009984)  req@ffff8e760d7eb850 x1678216578966464/t0(0) o3->181aa8b0-7375-4@10.50.13.10@o2ib2:592/0 lens 488/440 e 0 to 0 dl 1603145697 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:14:22 oak-io5-s2 kernel: Lustre: oak-OST0101: Bulk IO read error with 181aa8b0-7375-4 (at 10.50.13.10@o2ib2), client will retry: rc -110
Oct 19 15:14:22 oak-io5-s2 kernel: Lustre: Skipped 4 previous similar messages
Oct 19 15:14:22 oak-io5-s2 kernel: LustreError: 73006:0:(ldlm_lib.c:3294:target_bulk_io()) Skipped 5 previous similar messages
Oct 19 15:14:29 oak-io5-s2 kernel: Lustre: 28610:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1603145619/real 1603145669]  req@ffff8e7b06d10480 x1680988933818432/t0(0) o400->oak-MDT0002-lwp-OST00cc@10.0.2.51@o2ib5:12/10 lens 224/224 e 0 to 1 dl 1603145875 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1
Oct 19 15:14:29 oak-io5-s2 kernel: Lustre: 28610:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 121 previous similar messages
Oct 19 15:14:32 oak-io5-s2 kernel: LustreError: 347890:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 2072576(4169728)  req@ffff8e741bfeb850 x1678229260572864/t0(0) o3->a0cd7106-ff08-4@10.50.13.11@o2ib2:605/0 lens 488/440 e 0 to 0 dl 1603145710 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:14:32 oak-io5-s2 kernel: Lustre: oak-OST00e0: Bulk IO read error with a0cd7106-ff08-4 (at 10.50.13.11@o2ib2), client will retry: rc -110
Oct 19 15:14:32 oak-io5-s2 kernel: Lustre: Skipped 2 previous similar messages
Oct 19 15:14:32 oak-io5-s2 kernel: LustreError: 347890:0:(ldlm_lib.c:3294:target_bulk_io()) Skipped 4 previous similar messages
Oct 19 15:14:36 oak-io5-s2 kernel: LustreError: 401:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8e5a53bd7050 x1678111749827776/t0(0) o4->9dba092a-3435-4@10.50.15.10@o2ib2:617/0 lens 488/448 e 0 to 0 dl 1603145722 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:14:36 oak-io5-s2 kernel: Lustre: oak-OST00e7: Bulk IO write error with 9dba092a-3435-4 (at 10.50.15.10@o2ib2), client will retry: rc = -110
Oct 19 15:14:38 oak-io5-s2 kernel: LustreError: 63792:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8e2a17205850 x1672426044024832/t0(0) o4->09c51fad-0b0b-4@10.49.26.34@o2ib1:612/0 lens 488/448 e 0 to 0 dl 1603145717 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:14:38 oak-io5-s2 kernel: Lustre: oak-OST00ec: Bulk IO write error with 09c51fad-0b0b-4 (at 10.49.26.34@o2ib1), client will retry: rc = -110
Oct 19 15:14:42 oak-io5-s2 kernel: LustreError: 347790:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff8e741a288050 x1672426044023808/t0(0) o4->09c51fad-0b0b-4@10.49.26.34@o2ib1:598/0 lens 488/448 e 1 to 0 dl 1603145703 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:14:43 oak-io5-s2 kernel: LNet: 28355:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.2.51@o2ib5: 0 seconds
Oct 19 15:14:43 oak-io5-s2 kernel: LNet: 28355:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 192 previous similar messages
Oct 19 15:14:47 oak-io5-s2 kernel: Lustre: oak-OST0101: Bulk IO write error with 9dba092a-3435-4 (at 10.50.15.10@o2ib2), client will retry: rc = -110
Oct 19 15:14:47 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 15:14:52 oak-io5-s2 kernel: LustreError: 63792:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 2097152(2342912)  req@ffff8e7a4d7d7050 x1672282768241984/t0(0) o3->b09dcf48-669f-4@10.49.17.15@o2ib1:621/0 lens 488/440 e 0 to 0 dl 1603145726 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:14:52 oak-io5-s2 kernel: Lustre: oak-OST00f3: Bulk IO read error with e6bc8862-e330-4 (at 10.50.1.58@o2ib2), client will retry: rc -110
Oct 19 15:14:52 oak-io5-s2 kernel: Lustre: Skipped 4 previous similar messages
Oct 19 15:14:52 oak-io5-s2 kernel: LustreError: 63792:0:(ldlm_lib.c:3294:target_bulk_io()) Skipped 3 previous similar messages
Oct 19 15:14:53 oak-io5-s2 kernel: LustreError: 347794:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8ed35aa6a850 x1672623094118528/t0(0) o4->e6bc8862-e330-4@10.50.1.58@o2ib2:637/0 lens 504/448 e 0 to 0 dl 1603145742 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:14:53 oak-io5-s2 kernel: LustreError: 347794:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 1 previous similar message
Oct 19 15:14:56 oak-io5-s2 kernel: Lustre: oak-OST00f5: Client f659e999-70a6-4 (at 10.50.10.57@o2ib2) reconnecting
Oct 19 15:14:56 oak-io5-s2 kernel: Lustre: Skipped 9259 previous similar messages
Oct 19 15:14:57 oak-io5-s2 kernel: Lustre: oak-OST00fb: Bulk IO write error with 3af9e2b5-1b3d-4 (at 10.50.14.8@o2ib2), client will retry: rc = -110
Oct 19 15:14:57 oak-io5-s2 kernel: Lustre: Skipped 5 previous similar messages
Oct 19 15:15:02 oak-io5-s2 kernel: LustreError: 347814:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff8ece05a4d850 x1672435606287232/t0(0) o4->504c7155-aa0a-4@10.50.16.15@o2ib2:632/0 lens 488/448 e 0 to 0 dl 1603145737 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:15:02 oak-io5-s2 kernel: LustreError: 347814:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
Oct 19 15:15:07 oak-io5-s2 kernel: LustreError: 347823:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8ecdff2ce850 x1672201666795904/t0(0) o4->9ef17e7c-cbdd-4@10.50.9.43@o2ib2:626/0 lens 488/448 e 1 to 0 dl 1603145731 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:15:07 oak-io5-s2 kernel: LustreError: 347823:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 2 previous similar messages
Oct 19 15:15:17 oak-io5-s2 kernel: Lustre: oak-OST00ec: Bulk IO write error with 09c51fad-0b0b-4 (at 10.49.26.34@o2ib1), client will retry: rc = -110
Oct 19 15:15:17 oak-io5-s2 kernel: LustreError: 64883:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 0(516096)  req@ffff8ed1a0b6e850 x1678229262244288/t0(0) o3->a0cd7106-ff08-4@10.50.13.11@o2ib2:653/0 lens 488/440 e 0 to 0 dl 1603145758 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:15:17 oak-io5-s2 kernel: Lustre: oak-OST00d0: Bulk IO read error with a0cd7106-ff08-4 (at 10.50.13.11@o2ib2), client will retry: rc -110
Oct 19 15:15:17 oak-io5-s2 kernel: Lustre: Skipped 3 previous similar messages
Oct 19 15:15:17 oak-io5-s2 kernel: Lustre: Skipped 11 previous similar messages
Oct 19 15:15:22 oak-io5-s2 kernel: LustreError: 347846:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 0(516096)  req@ffff8ed9487b9050 x1678271368540352/t0(0) o3->57ecc9c0-d797-4@10.50.14.9@o2ib2:654/0 lens 488/440 e 0 to 0 dl 1603145759 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:15:22 oak-io5-s2 kernel: Lustre: oak-OST00fd: Bulk IO read error with 57ecc9c0-d797-4 (at 10.50.14.9@o2ib2), client will retry: rc -110
Oct 19 15:15:22 oak-io5-s2 kernel: Lustre: Skipped 3 previous similar messages
Oct 19 15:15:22 oak-io5-s2 kernel: LustreError: 347846:0:(ldlm_lib.c:3294:target_bulk_io()) Skipped 5 previous similar messages
Oct 19 15:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c6: Connection restored to  (at 10.50.3.2@o2ib2)
Oct 19 15:15:26 oak-io5-s2 kernel: Lustre: Skipped 25422 previous similar messages
Oct 19 15:15:30 oak-io5-s2 kernel: LustreError: 63586:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff8e7056e38850 x1678061800427968/t0(0) o3->c7058f34-a728-4@10.50.14.15@o2ib2:670/0 lens 488/440 e 0 to 0 dl 1603145775 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:15:30 oak-io5-s2 kernel: Lustre: oak-OST00fa: Bulk IO read error with c7058f34-a728-4 (at 10.50.14.15@o2ib2), client will retry: rc -110
Oct 19 15:15:30 oak-io5-s2 kernel: Lustre: Skipped 2 previous similar messages
Oct 19 15:15:50 oak-io5-s2 kernel: Lustre: oak-OST0100: Bulk IO read error with e6bc8862-e330-4 (at 10.50.1.58@o2ib2), client will retry: rc -110
Oct 19 15:15:50 oak-io5-s2 kernel: Lustre: oak-OST00d2: Bulk IO write error with 9ef17e7c-cbdd-4 (at 10.50.9.43@o2ib2), client will retry: rc = -110
Oct 19 15:15:50 oak-io5-s2 kernel: Lustre: Skipped 3 previous similar messages
Oct 19 15:15:52 oak-io5-s2 kernel: LustreError: 347835:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff8e6e30f23050 x1672324570096576/t0(0) o4->92eb3f89-d9ef-4@10.50.5.41@o2ib2:678/0 lens 488/448 e 0 to 0 dl 1603145783 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:15:52 oak-io5-s2 kernel: LustreError: 347835:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 12 previous similar messages
Oct 19 15:15:59 oak-io5-s2 kernel: LNet: 28355:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Timed out tx for 10.0.2.51@o2ib5: 3 seconds
Oct 19 15:15:59 oak-io5-s2 kernel: LNet: 28355:0:(o2iblnd_cb.c:3397:kiblnd_check_conns()) Skipped 356 previous similar messages
Oct 19 15:16:02 oak-io5-s2 kernel: LustreError: 347785:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 0(4194304)  req@ffff8e5a5d5f4850 x1672201666886336/t0(0) o3->9ef17e7c-cbdd-4@10.50.9.43@o2ib2:696/0 lens 488/440 e 0 to 0 dl 1603145801 ref 1 fl Interpret:/2/0 rc 0/0
Oct 19 15:16:02 oak-io5-s2 kernel: LustreError: 347785:0:(ldlm_lib.c:3294:target_bulk_io()) Skipped 5 previous similar messages
Oct 19 15:16:09 oak-io5-s2 kernel: LustreError: 347854:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8ed95a0d0850 x1672623094264832/t0(0) o4->e6bc8862-e330-4@10.50.1.58@o2ib2:715/0 lens 488/448 e 0 to 0 dl 1603145820 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:16:09 oak-io5-s2 kernel: LustreError: 347854:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 2 previous similar messages
Oct 19 15:16:11 oak-io5-s2 kernel: Lustre: oak-OST00c4: Client 0a221b6e-c1af-4 (at 10.49.17.14@o2ib1) reconnecting
Oct 19 15:16:11 oak-io5-s2 kernel: Lustre: Skipped 73227 previous similar messages
Oct 19 15:16:22 oak-io5-s2 kernel: LustreError: 347873:0:(ldlm_lib.c:3294:target_bulk_io()) @@@ truncated bulk READ 0(196608)  req@ffff8ed478e9e050 x1678271368549824/t0(0) o3->57ecc9c0-d797-4@10.50.14.9@o2ib2:715/0 lens 488/440 e 0 to 0 dl 1603145820 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:16:22 oak-io5-s2 kernel: Lustre: oak-OST00d4: Bulk IO read error with 57ecc9c0-d797-4 (at 10.50.14.9@o2ib2), client will retry: rc -110
Oct 19 15:16:22 oak-io5-s2 kernel: Lustre: Skipped 6 previous similar messages
Oct 19 15:17:12 oak-io5-s2 kernel: Lustre: 28621:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1603145576/real 1603145609]  req@ffff8e4b852bf500 x1680988932776320/t0(0) o400->oak-MDT0002-lwp-OST00fb@10.0.2.52@o2ib5:12/10 lens 224/224 e 0 to 1 dl 1603145832 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Oct 19 15:17:12 oak-io5-s2 kernel: Lustre: 28620:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1603145576/real 1603145609]  req@ffff8e4b852b8d80 x1680988932776256/t0(0) o400->oak-MDT0001-lwp-OST00fb@10.0.2.52@o2ib5:12/10 lens 224/224 e 0 to 1 dl 1603145832 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Oct 19 15:17:12 oak-io5-s2 kernel: Lustre: 28620:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 112 previous similar messages
Oct 19 15:17:12 oak-io5-s2 kernel: Lustre: 28621:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 39 previous similar messages
Oct 19 15:17:30 oak-io5-s2 kernel: Lustre: Evicted from MGS (at 10.0.2.51@o2ib5) after server handle changed from 0xe88ce0ce95a0fccd to 0xc71a30beeaf99dc5
Oct 19 15:20:14 oak-io5-s2 kernel: LustreError: 453747:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f4: cli c08cd4ef-d530-4 claims 4218880 GRANT, real grant 2744320
Oct 19 15:20:14 oak-io5-s2 kernel: LustreError: 453747:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 11 previous similar messages
Oct 19 15:23:46 oak-io5-s2 kernel: LustreError: 167-0: oak-MDT0001-lwp-OST00d0: This client was evicted by oak-MDT0001; in progress operations using this service will fail.
Oct 19 15:23:46 oak-io5-s2 kernel: LustreError: Skipped 69 previous similar messages
Oct 19 15:25:27 oak-io5-s2 kernel: Lustre: oak-OST00db: Connection restored to oak-MDT0002-mdtlov_UUID (at 10.0.2.51@o2ib5)
Oct 19 15:25:27 oak-io5-s2 kernel: Lustre: Skipped 114365 previous similar messages
Oct 19 15:26:42 oak-io5-s2 kernel: LustreError: 167-0: oak-MDT0002-lwp-OST010c: This client was evicted by oak-MDT0002; in progress operations using this service will fail.
Oct 19 15:26:42 oak-io5-s2 kernel: LustreError: Skipped 79 previous similar messages
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00c0: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6ce8380c00, cur 1603146423 expire 1603146273 last 1603146196
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: Skipped 241 previous similar messages
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00c0: deleting orphan objects from 0x3300000402:4137123 to 0x3300000402:4137153
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00c4: deleting orphan objects from 0x3400000402:4241773 to 0x3400000402:4241793
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00c3: deleting orphan objects from 0x33c0000401:3970124 to 0x33c0000401:3970145
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00c1: deleting orphan objects from 0x3340000402:4062119 to 0x3340000402:4062145
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00c6: deleting orphan objects from 0x3480000402:4338822 to 0x3480000402:4338849
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00c8: deleting orphan objects from 0x3500000401:4187540 to 0x3500000401:4187585
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00c9: deleting orphan objects from 0x3540000402:4223368 to 0x3540000402:4223393
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00c7: deleting orphan objects from 0x34c0000402:4243626 to 0x34c0000402:4243649
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00cc: deleting orphan objects from 0x3600000401:4688821 to 0x3600000401:4688865
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00ce: deleting orphan objects from 0x3680000401:4243798 to 0x3680000401:4243841
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00cd: deleting orphan objects from 0x3640000402:4678647 to 0x3640000402:4678689
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00d1: deleting orphan objects from 0x3740000402:4726549 to 0x3740000402:4726593
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00ca: deleting orphan objects from 0x3580000401:4709622 to 0x3580000401:4709665
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00d4: deleting orphan objects from 0x3800000402:4791864 to 0x3800000402:4791905
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00cf: deleting orphan objects from 0x36c0000402:4691617 to 0x36c0000402:4691649
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00c2: deleting orphan objects from 0x3380000402:3943824 to 0x3380000402:3943841
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00d5: deleting orphan objects from 0x3840000401:4605525 to 0x3840000401:4605569
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00d2: deleting orphan objects from 0x3780000402:4621015 to 0x3780000402:4621057
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00cb: deleting orphan objects from 0x35c0000402:4745337 to 0x35c0000402:4745377
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00d9: deleting orphan objects from 0x3940000402:4737366 to 0x3940000402:4737409
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00dc: deleting orphan objects from 0x3a00000402:4728549 to 0x3a00000402:4728577
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00d6: deleting orphan objects from 0x3880000401:4598808 to 0x3880000401:4598849
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00d8: deleting orphan objects from 0x3900000402:4730419 to 0x3900000402:4730465
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00d7: deleting orphan objects from 0x38c0000401:4571020 to 0x38c0000401:4571041
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00e4: deleting orphan objects from 0x3c00000401:4679884 to 0x3c00000401:4679905
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00dd: deleting orphan objects from 0x3a40000402:4679095 to 0x3a40000402:4679137
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00e8: deleting orphan objects from 0x3d00000402:4770274 to 0x3d00000402:4770305
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00df: deleting orphan objects from 0x3ac0000402:4785154 to 0x3ac0000402:4785185
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00ea: deleting orphan objects from 0x3d80000401:4705248 to 0x3d80000401:4705281
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00e6: deleting orphan objects from 0x3c80000401:4766076 to 0x3c80000401:4766113
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00de: deleting orphan objects from 0x3a80000401:4640601 to 0x3a80000401:4640641
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00ed: deleting orphan objects from 0x3e40000402:4765789 to 0x3e40000402:4765825
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00ee: deleting orphan objects from 0x3e80000402:4506974 to 0x3e80000402:4507009
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00e2: deleting orphan objects from 0x3b80000401:4657966 to 0x3b80000401:4657985
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00ec: deleting orphan objects from 0x3e00000401:4552059 to 0x3e00000401:4552097
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00f4: deleting orphan objects from 0x4000000401:5163069 to 0x4000000401:5163105
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00f3: deleting orphan objects from 0x3fc0000401:5093617 to 0x3fc0000401:5093633
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00e0: deleting orphan objects from 0x3b00000401:4648275 to 0x3b00000401:4648321
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00f5: deleting orphan objects from 0x4040000401:5114145 to 0x4040000401:5114177
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00ef: deleting orphan objects from 0x3ec0000402:4681364 to 0x3ec0000402:4681409
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00f6: deleting orphan objects from 0x4080000401:4981311 to 0x4080000401:4981345
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00f9: deleting orphan objects from 0x4140000401:5044347 to 0x4140000401:5044385
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00fb: deleting orphan objects from 0x41c0000401:5172033 to 0x41c0000401:5172065
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00f8: deleting orphan objects from 0x4100000402:5099860 to 0x4100000402:5099905
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00e1: deleting orphan objects from 0x3b40000401:4720122 to 0x3b40000401:4720161
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00fe: deleting orphan objects from 0x4280000401:5074872 to 0x4280000401:5074913
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00f1: deleting orphan objects from 0x3f40000401:4820945 to 0x3f40000401:4820961
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST0101: deleting orphan objects from 0x4340000402:4784944 to 0x4340000402:4784961
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST0100: deleting orphan objects from 0x4300000401:5088705 to 0x4300000401:5088737
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00e9: deleting orphan objects from 0x3d40000402:4807578 to 0x3d40000402:4807617
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00ff: deleting orphan objects from 0x42c0000402:5159582 to 0x42c0000402:5159617
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST0103: deleting orphan objects from 0x43c0000401:5107979 to 0x43c0000401:5108001
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00f2: deleting orphan objects from 0x3f80000401:5156290 to 0x3f80000401:5156321
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST0106: deleting orphan objects from 0x4480000401:4868670 to 0x4480000401:4868705
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST010b: deleting orphan objects from 0x45c0000401:4747005 to 0x45c0000401:4747041
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST0107: deleting orphan objects from 0x44c0000401:4405910 to 0x44c0000401:4405953
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00eb: deleting orphan objects from 0x3dc0000402:4786742 to 0x3dc0000402:4786785
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST0109: deleting orphan objects from 0x4540000402:4898944 to 0x4540000402:4898977
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00fc: deleting orphan objects from 0x4200000401:5063364 to 0x4200000401:5063393
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST010c: deleting orphan objects from 0x4600000402:4823397 to 0x4600000402:4823425
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00fd: deleting orphan objects from 0x4240000402:5211421 to 0x4240000402:5211457
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST010a: deleting orphan objects from 0x4580000401:4796578 to 0x4580000401:4796609
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST0102: deleting orphan objects from 0x4380000402:5086559 to 0x4380000402:5086593
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST010f: deleting orphan objects from 0x46c0000401:4837175 to 0x46c0000401:4837217
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00c5: deleting orphan objects from 0x3440000402:4073356 to 0x3440000402:4073377
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST0108: deleting orphan objects from 0x4500000401:4890242 to 0x4500000401:4890273
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00f7: deleting orphan objects from 0x40c0000401:4959113 to 0x40c0000401:4959137
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00d3: deleting orphan objects from 0x37c0000401:4720046 to 0x37c0000401:4720065
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00d0: deleting orphan objects from 0x3700000401:4732833 to 0x3700000401:4732865
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00da: deleting orphan objects from 0x3980000402:4782040 to 0x3980000402:4782081
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00e5: deleting orphan objects from 0x3c40000401:4516169 to 0x3c40000401:4516193
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00e7: deleting orphan objects from 0x3cc0000401:4777918 to 0x3cc0000401:4777953
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST0104: deleting orphan objects from 0x4400000401:5081604 to 0x4400000401:5081633
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00fa: deleting orphan objects from 0x4180000402:5132197 to 0x4180000402:5132225
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST010d: deleting orphan objects from 0x4640000402:4874449 to 0x4640000402:4874465
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST0105: deleting orphan objects from 0x4440000402:5120349 to 0x4440000402:5120385
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00f0: deleting orphan objects from 0x3f00000402:4683269 to 0x3f00000402:4683297
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00e3: deleting orphan objects from 0x3bc0000402:4696335 to 0x3bc0000402:4696353
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST00db: deleting orphan objects from 0x39c0000401:4501811 to 0x39c0000401:4501857
Oct 19 15:27:03 oak-io5-s2 kernel: Lustre: oak-OST010e: deleting orphan objects from 0x4680000401:4777962 to 0x4680000401:4777985
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00c2: deleting orphan objects from 0x3380000400:17839598 to 0x3380000400:17839617
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00c6: deleting orphan objects from 0x3480000400:19655251 to 0x3480000400:19655297
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00c0: deleting orphan objects from 0x3300000400:18552755 to 0x3300000400:18552801
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00ca: deleting orphan objects from 0x3580000402:18064394 to 0x3580000402:18064417
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00c8: deleting orphan objects from 0x3500000402:18722670 to 0x3500000402:18722689
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00d4: deleting orphan objects from 0x3800000400:18239905 to 0x3800000400:18239937
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00d9: deleting orphan objects from 0x3940000400:18227674 to 0x3940000400:18227713
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00c4: deleting orphan objects from 0x3400000400:18945970 to 0x3400000400:18945985
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00c9: deleting orphan objects from 0x3540000400:18879077 to 0x3540000400:18879105
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00d1: deleting orphan objects from 0x3740000400:18166474 to 0x3740000400:18166497
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00c3: deleting orphan objects from 0x33c0000402:17921352 to 0x33c0000402:17921377
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00e0: deleting orphan objects from 0x3b00000402:17585322 to 0x3b00000402:17585345
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00dd: deleting orphan objects from 0x3a40000400:17878397 to 0x3a40000400:17878433
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00d2: deleting orphan objects from 0x3780000400:17351813 to 0x3780000400:17351841
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00e7: deleting orphan objects from 0x3cc0000402:18413460 to 0x3cc0000402:18413505
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00db: deleting orphan objects from 0x39c0000402:17373620 to 0x39c0000402:17373665
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00ed: deleting orphan objects from 0x3e40000400:18322461 to 0x3e40000400:18322497
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00c7: deleting orphan objects from 0x34c0000400:18986253 to 0x34c0000400:18986273
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00c1: deleting orphan objects from 0x3340000400:18414222 to 0x3340000400:18414241
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00ee: deleting orphan objects from 0x3e80000400:17434470 to 0x3e80000400:17434497
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00d6: deleting orphan objects from 0x3880000402:17827192 to 0x3880000402:17827233
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00d8: deleting orphan objects from 0x3900000400:18096549 to 0x3900000400:18096577
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00f2: deleting orphan objects from 0x3f80000402:19832917 to 0x3f80000402:19832961
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00d7: deleting orphan objects from 0x38c0000402:17460068 to 0x38c0000402:17460097
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00f0: deleting orphan objects from 0x3f00000400:18036977 to 0x3f00000400:18036993
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00ce: deleting orphan objects from 0x3680000402:15256069 to 0x3680000402:15256097
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00f7: deleting orphan objects from 0x40c0000402:18835004 to 0x40c0000402:18835041
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00e5: deleting orphan objects from 0x3c40000402:17224907 to 0x3c40000402:17224929
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00fa: deleting orphan objects from 0x4180000400:19619842 to 0x4180000400:19619873
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00da: deleting orphan objects from 0x3980000400:18424225 to 0x3980000400:18424289
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00e2: deleting orphan objects from 0x3b80000402:17932531 to 0x3b80000402:17932577
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00df: deleting orphan objects from 0x3ac0000400:18404015 to 0x3ac0000400:18404033
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST0100: deleting orphan objects from 0x4300000402:19568035 to 0x4300000402:19568065
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST0102: deleting orphan objects from 0x4380000400:19338834 to 0x4380000400:19338849
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00e6: deleting orphan objects from 0x3c80000402:18371655 to 0x3c80000402:18371681
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST0106: deleting orphan objects from 0x4480000402:19243470 to 0x4480000402:19243489
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00f8: deleting orphan objects from 0x4100000400:19472565 to 0x4100000400:19472609
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00dc: deleting orphan objects from 0x3a00000400:17928531 to 0x3a00000400:17928577
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00eb: deleting orphan objects from 0x3dc0000400:18247163 to 0x3dc0000400:18247201
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST0107: deleting orphan objects from 0x44c0000402:16877127 to 0x44c0000402:16877153
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST010b: deleting orphan objects from 0x45c0000402:18631089 to 0x45c0000402:18631105
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00e3: deleting orphan objects from 0x3bc0000400:18072815 to 0x3bc0000400:18072833
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST0108: deleting orphan objects from 0x4500000402:19396474 to 0x4500000402:19396513
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00e4: deleting orphan objects from 0x3c00000402:18000745 to 0x3c00000402:18000769
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST010f: deleting orphan objects from 0x46c0000402:19105096 to 0x46c0000402:19105121
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00ef: deleting orphan objects from 0x3ec0000400:17859503 to 0x3ec0000400:17859521
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00de: deleting orphan objects from 0x3a80000402:17582832 to 0x3a80000402:17582849
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00f4: deleting orphan objects from 0x4000000402:19675173 to 0x4000000402:19675201
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST0109: deleting orphan objects from 0x4540000400:19274453 to 0x4540000400:19274497
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00fb: deleting orphan objects from 0x41c0000402:19778658 to 0x41c0000402:19778689
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00ea: deleting orphan objects from 0x3d80000402:18196286 to 0x3d80000402:18196321
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00f3: deleting orphan objects from 0x3fc0000402:19543868 to 0x3fc0000402:19543905
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00ec: deleting orphan objects from 0x3e00000402:17490865 to 0x3e00000402:17490881
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00f9: deleting orphan objects from 0x4140000402:19289203 to 0x4140000402:19289249
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00d0: deleting orphan objects from 0x3700000402:18278555 to 0x3700000402:18278593
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00fe: deleting orphan objects from 0x4280000402:19511379 to 0x4280000402:19511425
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00f6: deleting orphan objects from 0x4080000402:19113973 to 0x4080000402:19114017
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00ff: deleting orphan objects from 0x42c0000400:19885450 to 0x42c0000400:19885473
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00f5: deleting orphan objects from 0x4040000402:19555849 to 0x4040000402:19555873
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST0101: deleting orphan objects from 0x4340000400:18056595 to 0x4340000400:18056641
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST0104: deleting orphan objects from 0x4400000402:19637273 to 0x4400000402:19637313
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST0105: deleting orphan objects from 0x4440000400:19679377 to 0x4440000400:19679393
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST0103: deleting orphan objects from 0x43c0000402:19656990 to 0x43c0000402:19657025
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST010a: deleting orphan objects from 0x4580000402:18809232 to 0x4580000402:18809249
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00cb: deleting orphan objects from 0x35c0000400:18297292 to 0x35c0000400:18297313
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00e8: deleting orphan objects from 0x3d00000400:18402853 to 0x3d00000400:18402881
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00d3: deleting orphan objects from 0x37c0000402:18165620 to 0x37c0000402:18165665
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00e9: deleting orphan objects from 0x3d40000400:18505257 to 0x3d40000400:18505281
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00fd: deleting orphan objects from 0x4240000400:20033993 to 0x4240000400:20034017
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST010c: deleting orphan objects from 0x4600000400:18931241 to 0x4600000400:18931265
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST010e: deleting orphan objects from 0x4680000402:18930661 to 0x4680000402:18930689
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00fc: deleting orphan objects from 0x4200000402:19256125 to 0x4200000402:19256161
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00c5: deleting orphan objects from 0x3440000400:18419430 to 0x3440000400:18419457
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00cc: deleting orphan objects from 0x3600000402:17764464 to 0x3600000402:17764481
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00cd: deleting orphan objects from 0x3640000400:18033369 to 0x3640000400:18033409
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00cf: deleting orphan objects from 0x36c0000400:17946523 to 0x36c0000400:17946561
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00d5: deleting orphan objects from 0x3840000400:16224235 to 0x3840000400:16224257
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00e1: deleting orphan objects from 0x3b40000402:18187320 to 0x3b40000402:18187361
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST00f1: deleting orphan objects from 0x3f40000402:18405344 to 0x3f40000402:18405377
Oct 19 15:27:13 oak-io5-s2 kernel: Lustre: oak-OST010d: deleting orphan objects from 0x4640000400:19302101 to 0x4640000400:19302145
Oct 19 15:29:12 oak-io5-s2 kernel: Lustre: 68783:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1603146545/real 1603146545]  req@ffff8e7af7528d80 x1680988945139072/t0(0) o105->oak-OST00e4@10.210.9.195@tcp1:15/16 lens 360/224 e 0 to 1 dl 1603146552 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Oct 19 15:29:12 oak-io5-s2 kernel: Lustre: 68783:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Oct 19 15:29:27 oak-io5-s2 kernel: LustreError: 410:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(4194304)  req@ffff8e79542cb050 x1678199521153056/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:743/0 lens 488/448 e 0 to 0 dl 1603146603 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:29:27 oak-io5-s2 kernel: Lustre: oak-OST00e8: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 15:29:27 oak-io5-s2 kernel: Lustre: Skipped 28 previous similar messages
Oct 19 15:29:27 oak-io5-s2 kernel: LustreError: 410:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 49 previous similar messages
Oct 19 15:29:47 oak-io5-s2 kernel: LustreError: 63811:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(4194304)  req@ffff8ed95084c050 x1678199521230224/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:11/0 lens 488/448 e 0 to 0 dl 1603146626 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:29:47 oak-io5-s2 kernel: Lustre: oak-OST00e5: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 15:29:47 oak-io5-s2 kernel: Lustre: Skipped 42 previous similar messages
Oct 19 15:29:47 oak-io5-s2 kernel: LustreError: 63811:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
Oct 19 15:30:08 oak-io5-s2 kernel: Lustre: oak-OST00e8: Client c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1) reconnecting
Oct 19 15:30:08 oak-io5-s2 kernel: Lustre: Skipped 52636 previous similar messages
Oct 19 15:30:08 oak-io5-s2 kernel: LustreError: 347821:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8ecf45270850 x1678199521485776/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:40/0 lens 488/448 e 0 to 0 dl 1603146655 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:30:08 oak-io5-s2 kernel: LustreError: 347821:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 17 previous similar messages
Oct 19 15:30:08 oak-io5-s2 kernel: Lustre: oak-OST00e8: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 15:30:08 oak-io5-s2 kernel: Lustre: Skipped 3 previous similar messages
Oct 19 15:30:18 oak-io5-s2 kernel: LustreError: 94138:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8ece0f613850 x1678199521506896/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:46/0 lens 488/448 e 0 to 0 dl 1603146661 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:30:18 oak-io5-s2 kernel: LustreError: 94138:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 3 previous similar messages
Oct 19 15:30:37 oak-io5-s2 kernel: LustreError: 347844:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(114938)  req@ffff8e7a3b107850 x1678199521193424/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:59/0 lens 488/448 e 0 to 0 dl 1603146674 ref 1 fl Interpret:/2/0 rc 0/0
Oct 19 15:30:37 oak-io5-s2 kernel: LustreError: 347844:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 14 previous similar messages
Oct 19 15:30:52 oak-io5-s2 kernel: Lustre: oak-OST00f8: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 15:30:52 oak-io5-s2 kernel: Lustre: Skipped 37 previous similar messages
Oct 19 15:31:13 oak-io5-s2 kernel: Lustre: oak-OST00e6: Client c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1) reconnecting
Oct 19 15:31:13 oak-io5-s2 kernel: Lustre: Skipped 30 previous similar messages
Oct 19 15:31:13 oak-io5-s2 kernel: LustreError: 109112:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8e740fad4850 x1678199521912384/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:107/0 lens 488/448 e 0 to 0 dl 1603146722 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:31:13 oak-io5-s2 kernel: LustreError: 109112:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 4 previous similar messages
Oct 19 15:31:22 oak-io5-s2 kernel: LustreError: 67803:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(670616)  req@ffff8e6cc5cb3050 x1678199521851920/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:105/0 lens 488/448 e 0 to 0 dl 1603146720 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:31:22 oak-io5-s2 kernel: LustreError: 67803:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 22 previous similar messages
Oct 19 15:31:52 oak-io5-s2 kernel: Lustre: oak-OST00fe: Client c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1) reconnecting
Oct 19 15:31:52 oak-io5-s2 kernel: Lustre: Skipped 17 previous similar messages
Oct 19 15:43:16 oak-io5-s2 kernel: Lustre: oak-OST010a: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 15:43:16 oak-io5-s2 kernel: Lustre: Skipped 1128 previous similar messages
Oct 19 15:46:19 oak-io5-s2 kernel: LustreError: 347846:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00fb: cli af146c4d-8b0e-699a-c653-e467ecdb4220 claims 4218880 GRANT, real grant 700416
Oct 19 15:46:19 oak-io5-s2 kernel: LustreError: 347846:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 53 previous similar messages
Oct 19 15:47:03 oak-io5-s2 kernel: Lustre: oak-OST0101: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6cc3c59400, cur 1603147623 expire 1603147473 last 1603147396
Oct 19 15:47:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 15:51:52 oak-io5-s2 kernel: LustreError: 109101:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 822511(1871087)  req@ffff8e740e2c0050 x1678199533754752/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:581/0 lens 488/448 e 0 to 0 dl 1603147951 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:51:52 oak-io5-s2 kernel: Lustre: oak-OST00d2: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 15:51:52 oak-io5-s2 kernel: Lustre: Skipped 24 previous similar messages
Oct 19 15:51:52 oak-io5-s2 kernel: LustreError: 109101:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 25 previous similar messages
Oct 19 15:52:35 oak-io5-s2 kernel: Lustre: oak-OST00ea: Client c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1) reconnecting
Oct 19 15:52:35 oak-io5-s2 kernel: Lustre: Skipped 8 previous similar messages
Oct 19 15:52:36 oak-io5-s2 kernel: LustreError: 64882:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8ecdf773f850 x1678199533841152/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:636/0 lens 488/448 e 0 to 0 dl 1603148006 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:52:36 oak-io5-s2 kernel: LustreError: 64882:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 13 previous similar messages
Oct 19 15:52:36 oak-io5-s2 kernel: Lustre: oak-OST00d7: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 15:52:36 oak-io5-s2 kernel: Lustre: Skipped 23 previous similar messages
Oct 19 15:52:38 oak-io5-s2 kernel: LustreError: 63792:0:(ldlm_lib.c:3228:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff8e3c7eb9d850 x1678199533847680/t0(0) o3->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:640/0 lens 488/440 e 0 to 0 dl 1603148010 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 15:52:38 oak-io5-s2 kernel: Lustre: oak-OST0100: Bulk IO read error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc -107
Oct 19 15:52:38 oak-io5-s2 kernel: LustreError: 63792:0:(ldlm_lib.c:3228:target_bulk_io()) Skipped 1 previous similar message
Oct 19 16:03:16 oak-io5-s2 kernel: Lustre: oak-OST010a: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 16:03:16 oak-io5-s2 kernel: Lustre: oak-OST010f: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 16:03:16 oak-io5-s2 kernel: Lustre: Skipped 90 previous similar messages
Oct 19 16:03:16 oak-io5-s2 kernel: Lustre: Skipped 8 previous similar messages
Oct 19 16:07:03 oak-io5-s2 kernel: Lustre: oak-OST00cf: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7a4e9db000, cur 1603148823 expire 1603148673 last 1603148596
Oct 19 16:07:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 16:11:44 oak-io5-s2 kernel: LustreError: 67803:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00cf: cli 68e9ac59-96c5-4 claims 4218880 GRANT, real grant 2744320
Oct 19 16:11:44 oak-io5-s2 kernel: LustreError: 67803:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 1 previous similar message
Oct 19 16:11:46 oak-io5-s2 kernel: LustreError: 347813:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00cf: cli 68e9ac59-96c5-4 claims 4218880 GRANT, real grant 0
Oct 19 16:11:46 oak-io5-s2 kernel: LustreError: 347813:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 1 previous similar message
Oct 19 16:11:50 oak-io5-s2 kernel: LustreError: 64587:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00cf: cli 68e9ac59-96c5-4 claims 4218880 GRANT, real grant 0
Oct 19 16:11:50 oak-io5-s2 kernel: LustreError: 64587:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 2 previous similar messages
Oct 19 16:11:54 oak-io5-s2 kernel: LustreError: 347844:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00cf: cli 68e9ac59-96c5-4 claims 4218880 GRANT, real grant 0
Oct 19 16:11:54 oak-io5-s2 kernel: LustreError: 347844:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 3 previous similar messages
Oct 19 16:12:04 oak-io5-s2 kernel: LustreError: 67796:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00cf: cli 68e9ac59-96c5-4 claims 4218880 GRANT, real grant 0
Oct 19 16:12:04 oak-io5-s2 kernel: LustreError: 67796:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 8 previous similar messages
Oct 19 16:12:24 oak-io5-s2 kernel: LustreError: 67802:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00cf: cli 68e9ac59-96c5-4 claims 4218880 GRANT, real grant 0
Oct 19 16:12:24 oak-io5-s2 kernel: LustreError: 67802:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 16 previous similar messages
Oct 19 16:21:47 oak-io5-s2 kernel: LustreError: 347889:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00cf: cli 68e9ac59-96c5-4 claims 28672 GRANT, real grant 0
Oct 19 16:21:47 oak-io5-s2 kernel: LustreError: 347889:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 7 previous similar messages
Oct 19 16:23:15 oak-io5-s2 kernel: Lustre: oak-OST0101: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 16:23:15 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 19 16:24:52 oak-io5-s2 kernel: LustreError: 109106:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00cf: cli 68e9ac59-96c5-4 claims 2023424 GRANT, real grant 0
Oct 19 16:27:02 oak-io5-s2 kernel: Lustre: oak-OST00ed: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7a09be2c00, cur 1603150022 expire 1603149872 last 1603149795
Oct 19 16:27:02 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 16:29:01 oak-io5-s2 kernel: LustreError: 347877:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00cf: cli 68e9ac59-96c5-4 claims 28672 GRANT, real grant 0
Oct 19 16:43:16 oak-io5-s2 kernel: Lustre: oak-OST00e9: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 16:43:16 oak-io5-s2 kernel: Lustre: Skipped 77 previous similar messages
Oct 19 16:47:03 oak-io5-s2 kernel: Lustre: oak-OST00c0: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e741bf23c00, cur 1603151223 expire 1603151073 last 1603150996
Oct 19 16:47:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 16:55:12 oak-io5-s2 kernel: Lustre: 120536:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1603151705/real 1603151705]  req@ffff8e5b66c25e80 x1680989049052032/t0(0) o104->oak-OST010d@10.210.9.195@tcp1:15/16 lens 296/224 e 0 to 1 dl 1603151712 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Oct 19 17:03:15 oak-io5-s2 kernel: Lustre: oak-OST010c: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 17:03:15 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 17:07:02 oak-io5-s2 kernel: Lustre: oak-OST00ed: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e740c400800, cur 1603152422 expire 1603152272 last 1603152195
Oct 19 17:07:02 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 17:23:16 oak-io5-s2 kernel: Lustre: oak-OST010c: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 17:23:16 oak-io5-s2 kernel: Lustre: oak-OST0101: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 17:23:16 oak-io5-s2 kernel: Lustre: Skipped 68 previous similar messages
Oct 19 17:23:16 oak-io5-s2 kernel: Lustre: Skipped 8 previous similar messages
Oct 19 17:27:03 oak-io5-s2 kernel: Lustre: oak-OST00cf: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7a02b72c00, cur 1603153623 expire 1603153473 last 1603153396
Oct 19 17:27:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 17:32:27 oak-io5-s2 kernel: LustreError: 64577:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1048576(3090262)  req@ffff8ed4789be850 x1678199623322144/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:565/0 lens 488/448 e 1 to 0 dl 1603153975 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 17:32:27 oak-io5-s2 kernel: Lustre: oak-OST010c: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 17:32:27 oak-io5-s2 kernel: Lustre: Skipped 7 previous similar messages
Oct 19 17:32:27 oak-io5-s2 kernel: LustreError: 64577:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 10 previous similar messages
Oct 19 17:32:40 oak-io5-s2 kernel: Lustre: oak-OST00c6: Client c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1) reconnecting
Oct 19 17:32:40 oak-io5-s2 kernel: Lustre: Skipped 21 previous similar messages
Oct 19 17:33:01 oak-io5-s2 kernel: Lustre: oak-OST010c: Client c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1) reconnecting
Oct 19 17:33:03 oak-io5-s2 kernel: LustreError: 64887:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff8ed92ec3f050 x1678199623400016/t0(0) o3->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:627/0 lens 488/440 e 0 to 0 dl 1603154037 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 17:33:03 oak-io5-s2 kernel: LustreError: 64887:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 7 previous similar messages
Oct 19 17:33:03 oak-io5-s2 kernel: Lustre: oak-OST00fb: Bulk IO read error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc -110
Oct 19 17:33:03 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 17:33:04 oak-io5-s2 kernel: Lustre: oak-OST00fa: Client c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1) reconnecting
Oct 19 17:33:04 oak-io5-s2 kernel: Lustre: Skipped 5 previous similar messages
Oct 19 17:39:02 oak-io5-s2 kernel: LustreError: 412:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1048576(4194304)  req@ffff8e6cdc1df850 x1678199625930992/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:220/0 lens 488/448 e 0 to 0 dl 1603154385 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 17:39:02 oak-io5-s2 kernel: Lustre: oak-OST00ec: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 17:39:02 oak-io5-s2 kernel: Lustre: Skipped 10 previous similar messages
Oct 19 17:39:02 oak-io5-s2 kernel: LustreError: 412:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 14 previous similar messages
Oct 19 17:39:04 oak-io5-s2 kernel: Lustre: oak-OST00d4: Client c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1) reconnecting
Oct 19 17:39:04 oak-io5-s2 kernel: Lustre: oak-OST00e3: Connection restored to  (at 10.210.9.195@tcp1)
Oct 19 17:39:04 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 19 17:39:04 oak-io5-s2 kernel: Lustre: Skipped 5 previous similar messages
Oct 19 17:39:05 oak-io5-s2 kernel: LustreError: 347891:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8ed26f3f0850 x1678199626303040/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:234/0 lens 488/448 e 0 to 0 dl 1603154399 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 17:39:49 oak-io5-s2 kernel: Lustre: oak-OST00d5: Client c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1) reconnecting
Oct 19 17:47:03 oak-io5-s2 kernel: Lustre: oak-OST00cf: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7a7eba4400, cur 1603154823 expire 1603154673 last 1603154596
Oct 19 17:47:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 17:50:48 oak-io5-s2 kernel: Lustre: oak-OST0105: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 17:50:48 oak-io5-s2 kernel: Lustre: Skipped 85 previous similar messages
Oct 19 18:03:16 oak-io5-s2 kernel: Lustre: oak-OST010a: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 18:03:16 oak-io5-s2 kernel: Lustre: Skipped 8 previous similar messages
Oct 19 18:07:03 oak-io5-s2 kernel: Lustre: oak-OST00c0: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e760a7b3c00, cur 1603156023 expire 1603155873 last 1603155796
Oct 19 18:07:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 18:12:08 oak-io5-s2 kernel: LustreError: 347883:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0106: cli b5c06976-44ba-4 claims 4218880 GRANT, real grant 4141056
Oct 19 18:14:27 oak-io5-s2 kernel: LustreError: 347804:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff8e2442598050 x1678199666428400/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:77/0 lens 488/448 e 0 to 0 dl 1603156507 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 18:14:27 oak-io5-s2 kernel: Lustre: oak-OST00fe: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 18:14:27 oak-io5-s2 kernel: Lustre: Skipped 15 previous similar messages
Oct 19 18:14:27 oak-io5-s2 kernel: LustreError: 347804:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
Oct 19 18:15:18 oak-io5-s2 kernel: Lustre: oak-OST00fe: Client c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1) reconnecting
Oct 19 18:15:18 oak-io5-s2 kernel: Lustre: Skipped 3 previous similar messages
Oct 19 18:15:18 oak-io5-s2 kernel: Lustre: oak-OST00fe: Connection restored to  (at 10.210.9.195@tcp1)
Oct 19 18:15:18 oak-io5-s2 kernel: Lustre: Skipped 70 previous similar messages
Oct 19 18:15:23 oak-io5-s2 kernel: Lustre: oak-OST00e9: Client c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1) reconnecting
Oct 19 18:15:23 oak-io5-s2 kernel: LustreError: 63819:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8e74173d4850 x1678199667904224/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:138/0 lens 488/448 e 0 to 0 dl 1603156568 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 18:15:23 oak-io5-s2 kernel: Lustre: oak-OST00e9: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 18:15:23 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 18:15:23 oak-io5-s2 kernel: LustreError: 63819:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 1 previous similar message
Oct 19 18:16:02 oak-io5-s2 kernel: LustreError: 347857:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff8ecdca7fc850 x1678199668921264/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:171/0 lens 488/448 e 0 to 0 dl 1603156601 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 18:16:02 oak-io5-s2 kernel: Lustre: oak-OST00f3: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 18:16:02 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 18:16:02 oak-io5-s2 kernel: LustreError: 347857:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
Oct 19 18:16:09 oak-io5-s2 kernel: Lustre: 420253:0:(client.c:2133:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1603156562/real 1603156562]  req@ffff8eafd9a04800 x1680989150674496/t0(0) o104->oak-OST00f3@10.210.9.195@tcp1:15/16 lens 296/224 e 0 to 1 dl 1603156569 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
Oct 19 18:16:09 oak-io5-s2 kernel: Lustre: 420253:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Oct 19 18:16:12 oak-io5-s2 kernel: LustreError: 94625:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff8ed21db52050 x1678199669013472/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:184/0 lens 488/448 e 0 to 0 dl 1603156614 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 18:16:12 oak-io5-s2 kernel: Lustre: oak-OST00c3: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 18:16:12 oak-io5-s2 kernel: Lustre: Skipped 1 previous similar message
Oct 19 18:16:12 oak-io5-s2 kernel: LustreError: 94625:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
Oct 19 18:16:51 oak-io5-s2 kernel: Lustre: oak-OST00f3: Client c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1) reconnecting
Oct 19 18:16:59 oak-io5-s2 kernel: LustreError: 416761:0:(ldlm_lib.c:3279:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8e7a4c65f050 x1678199670134960/t0(0) o4->c50f4c48-a8d0-2d5f-ff90-7efef0b098e9@10.210.9.195@tcp1:231/0 lens 488/448 e 0 to 0 dl 1603156661 ref 1 fl Interpret:/0/0 rc 0/0
Oct 19 18:16:59 oak-io5-s2 kernel: Lustre: oak-OST00e9: Bulk IO write error with c50f4c48-a8d0-2d5f-ff90-7efef0b098e9 (at 10.210.9.195@tcp1), client will retry: rc = -110
Oct 19 18:16:59 oak-io5-s2 kernel: Lustre: Skipped 3 previous similar messages
Oct 19 18:16:59 oak-io5-s2 kernel: LustreError: 416761:0:(ldlm_lib.c:3279:target_bulk_io()) Skipped 2 previous similar messages
Oct 19 18:25:27 oak-io5-s2 kernel: Lustre: oak-OST010b: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 18:25:27 oak-io5-s2 kernel: Lustre: Skipped 66 previous similar messages
Oct 19 18:27:03 oak-io5-s2 kernel: Lustre: oak-OST00cf: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7a44304400, cur 1603157223 expire 1603157073 last 1603156996
Oct 19 18:27:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 18:43:16 oak-io5-s2 kernel: Lustre: oak-OST0103: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 18:43:16 oak-io5-s2 kernel: Lustre: Skipped 26 previous similar messages
Oct 19 18:47:03 oak-io5-s2 kernel: Lustre: oak-OST0101: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6e20885c00, cur 1603158423 expire 1603158273 last 1603158196
Oct 19 18:47:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 19:03:15 oak-io5-s2 kernel: Lustre: oak-OST010a: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 19:03:15 oak-io5-s2 kernel: Lustre: Skipped 75 previous similar messages
Oct 19 19:07:02 oak-io5-s2 kernel: Lustre: oak-OST00ed: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e740fb84400, cur 1603159622 expire 1603159472 last 1603159395
Oct 19 19:07:02 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 19:23:15 oak-io5-s2 kernel: Lustre: oak-OST010f: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 19:23:15 oak-io5-s2 kernel: Lustre: Skipped 77 previous similar messages
Oct 19 19:23:36 oak-io5-s2 kernel: LustreError: 84344:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00e1: cli a31e2f1f-00b5-4 claims 4231168 GRANT, real grant 4173824
Oct 19 19:23:36 oak-io5-s2 kernel: LustreError: 84344:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 1 previous similar message
Oct 19 19:27:02 oak-io5-s2 kernel: Lustre: oak-OST00e9: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6ce35c6c00, cur 1603160822 expire 1603160672 last 1603160595
Oct 19 19:27:02 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 19:43:15 oak-io5-s2 kernel: Lustre: oak-OST010c: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 19:43:15 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 19 19:47:02 oak-io5-s2 kernel: Lustre: oak-OST00cf: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6cc8935400, cur 1603162022 expire 1603161872 last 1603161795
Oct 19 19:47:02 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 20:03:16 oak-io5-s2 kernel: Lustre: oak-OST010a: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 20:03:16 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 19 20:07:03 oak-io5-s2 kernel: Lustre: oak-OST00cf: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e705d698800, cur 1603163223 expire 1603163073 last 1603162996
Oct 19 20:07:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 20:23:16 oak-io5-s2 kernel: Lustre: oak-OST010c: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 20:23:16 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 20:27:03 oak-io5-s2 kernel: Lustre: oak-OST00c0: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e760cf3d000, cur 1603164423 expire 1603164273 last 1603164196
Oct 19 20:27:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 20:43:16 oak-io5-s2 kernel: Lustre: oak-OST010f: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 20:43:16 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 20:47:03 oak-io5-s2 kernel: Lustre: oak-OST00ed: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e740c51ac00, cur 1603165623 expire 1603165473 last 1603165396
Oct 19 20:47:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 21:03:15 oak-io5-s2 kernel: Lustre: oak-OST010f: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 21:03:15 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 21:07:02 oak-io5-s2 kernel: Lustre: oak-OST00ed: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7a02823400, cur 1603166822 expire 1603166672 last 1603166595
Oct 19 21:07:02 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 21:13:54 oak-io5-s2 kernel: LustreError: 64585:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00e1: cli 7a6aa5e2-d91f-4 claims 4218880 GRANT, real grant 4083712
Oct 19 21:13:55 oak-io5-s2 kernel: LustreError: 67795:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00e1: cli 7a6aa5e2-d91f-4 claims 4218880 GRANT, real grant 0
Oct 19 21:13:57 oak-io5-s2 kernel: LustreError: 67795:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00e1: cli 7a6aa5e2-d91f-4 claims 4218880 GRANT, real grant 0
Oct 19 21:13:57 oak-io5-s2 kernel: LustreError: 67795:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 1 previous similar message
Oct 19 21:14:01 oak-io5-s2 kernel: LustreError: 347880:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00e1: cli 7a6aa5e2-d91f-4 claims 4218880 GRANT, real grant 0
Oct 19 21:14:01 oak-io5-s2 kernel: LustreError: 347880:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 2 previous similar messages
Oct 19 21:14:06 oak-io5-s2 kernel: LustreError: 109107:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00e1: cli 7a6aa5e2-d91f-4 claims 4218880 GRANT, real grant 0
Oct 19 21:14:06 oak-io5-s2 kernel: LustreError: 109107:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 4 previous similar messages
Oct 19 21:14:15 oak-io5-s2 kernel: LustreError: 347889:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00e1: cli 7a6aa5e2-d91f-4 claims 4218880 GRANT, real grant 0
Oct 19 21:14:15 oak-io5-s2 kernel: LustreError: 347889:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 8 previous similar messages
Oct 19 21:16:41 oak-io5-s2 kernel: LustreError: 64881:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00fa: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 4218880 GRANT, real grant 3530752
Oct 19 21:16:41 oak-io5-s2 kernel: LustreError: 64881:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 2 previous similar messages
Oct 19 21:20:29 oak-io5-s2 kernel: LNet: 31045:0:(o2iblnd_cb.c:2602:kiblnd_passive_connect()) Conn stale 10.0.2.244@o2ib5 version 12/12 incarnation 1594935264479742/1603167626943866
Oct 19 21:20:30 oak-io5-s2 kernel: Lustre: oak-OST00c3: Connection restored to  (at 10.0.2.244@o2ib5)
Oct 19 21:20:30 oak-io5-s2 kernel: Lustre: Skipped 108 previous similar messages
Oct 19 21:23:05 oak-io5-s2 kernel: Lustre: oak-OST0108: haven't heard from client 040e49ed-98ce-ddbd-0210-bfe3697c52b5 (at 10.0.2.244@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e776026e400, cur 1603167785 expire 1603167635 last 1603167558
Oct 19 21:23:05 oak-io5-s2 kernel: Lustre: Skipped 159 previous similar messages
Oct 19 21:30:47 oak-io5-s2 kernel: Lustre: oak-OST0105: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 21:30:47 oak-io5-s2 kernel: Lustre: Skipped 119 previous similar messages
Oct 19 21:34:34 oak-io5-s2 kernel: Lustre: oak-OST0105: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e761dfb3c00, cur 1603168474 expire 1603168324 last 1603168247
Oct 19 21:34:34 oak-io5-s2 kernel: Lustre: Skipped 158 previous similar messages
Oct 19 21:40:29 oak-io5-s2 kernel: LustreError: 347852:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00ca: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 4218880 GRANT, real grant 671744
Oct 19 21:40:29 oak-io5-s2 kernel: LustreError: 347852:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 19 previous similar messages
Oct 19 21:43:16 oak-io5-s2 kernel: Lustre: oak-OST010c: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 21:43:16 oak-io5-s2 kernel: Lustre: Skipped 9 previous similar messages
Oct 19 21:47:03 oak-io5-s2 kernel: Lustre: oak-OST0104: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7b0d5bd000, cur 1603169223 expire 1603169073 last 1603168996
Oct 19 21:48:33 oak-io5-s2 kernel: LustreError: 107339:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00ea: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 2035712 GRANT, real grant 540672
Oct 19 21:48:33 oak-io5-s2 kernel: LustreError: 107339:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 11 previous similar messages
Oct 19 21:49:35 oak-io5-s2 kernel: LustreError: 347889:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00ea: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 4218880 GRANT, real grant 0
Oct 19 21:53:16 oak-io5-s2 kernel: LustreError: 453749:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 3416064 GRANT, real grant 1421312
Oct 19 21:53:16 oak-io5-s2 kernel: LustreError: 453749:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 52 previous similar messages
Oct 19 21:56:00 oak-io5-s2 kernel: LustreError: 107345:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 4218880 GRANT, real grant 0
Oct 19 22:03:16 oak-io5-s2 kernel: Lustre: oak-OST00ed: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 22:03:16 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 19 22:07:03 oak-io5-s2 kernel: Lustre: oak-OST00c0: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7413a1e400, cur 1603170423 expire 1603170273 last 1603170196
Oct 19 22:07:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 22:23:11 oak-io5-s2 kernel: LustreError: 347820:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 258048 GRANT, real grant 0
Oct 19 22:23:11 oak-io5-s2 kernel: LustreError: 347820:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 7 previous similar messages
Oct 19 22:23:16 oak-io5-s2 kernel: Lustre: oak-OST010c: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 22:23:16 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 19 22:27:03 oak-io5-s2 kernel: Lustre: oak-OST0101: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7613320800, cur 1603171623 expire 1603171473 last 1603171396
Oct 19 22:27:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 22:41:27 oak-io5-s2 kernel: LustreError: 347818:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00ec: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 1249280 GRANT, real grant 0
Oct 19 22:41:27 oak-io5-s2 kernel: LustreError: 347818:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 4 previous similar messages
Oct 19 22:43:16 oak-io5-s2 kernel: Lustre: oak-OST010c: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 22:43:16 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 19 22:47:03 oak-io5-s2 kernel: Lustre: oak-OST0104: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6e29b5f400, cur 1603172823 expire 1603172673 last 1603172596
Oct 19 22:47:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 22:59:02 oak-io5-s2 kernel: LustreError: 109110:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0108: cli d6a7657a-e5f2-4 claims 4218880 GRANT, real grant 794624
Oct 19 22:59:11 oak-io5-s2 kernel: LustreError: 347821:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0108: cli d6a7657a-e5f2-4 claims 3502080 GRANT, real grant 0
Oct 19 22:59:15 oak-io5-s2 kernel: LustreError: 347874:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0108: cli d6a7657a-e5f2-4 claims 4218880 GRANT, real grant 0
Oct 19 22:59:15 oak-io5-s2 kernel: LustreError: 347874:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 1 previous similar message
Oct 19 22:59:25 oak-io5-s2 kernel: LustreError: 347887:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0108: cli d6a7657a-e5f2-4 claims 4218880 GRANT, real grant 0
Oct 19 22:59:25 oak-io5-s2 kernel: LustreError: 347887:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 1 previous similar message
Oct 19 22:59:30 oak-io5-s2 kernel: LustreError: 67800:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0108: cli d6a7657a-e5f2-4 claims 4218880 GRANT, real grant 0
Oct 19 22:59:41 oak-io5-s2 kernel: LustreError: 67798:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0108: cli d6a7657a-e5f2-4 claims 2695168 GRANT, real grant 0
Oct 19 22:59:41 oak-io5-s2 kernel: LustreError: 67798:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 1 previous similar message
Oct 19 23:00:02 oak-io5-s2 kernel: LustreError: 121388:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0108: cli d6a7657a-e5f2-4 claims 4218880 GRANT, real grant 0
Oct 19 23:00:02 oak-io5-s2 kernel: LustreError: 121388:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 2 previous similar messages
Oct 19 23:00:40 oak-io5-s2 kernel: LustreError: 347887:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0108: cli d6a7657a-e5f2-4 claims 4218880 GRANT, real grant 0
Oct 19 23:00:40 oak-io5-s2 kernel: LustreError: 347887:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 6 previous similar messages
Oct 19 23:01:55 oak-io5-s2 kernel: LustreError: 347793:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0108: cli d6a7657a-e5f2-4 claims 4218880 GRANT, real grant 0
Oct 19 23:01:55 oak-io5-s2 kernel: LustreError: 347793:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 8 previous similar messages
Oct 19 23:03:16 oak-io5-s2 kernel: Lustre: oak-OST010c: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 23:03:16 oak-io5-s2 kernel: Lustre: Skipped 77 previous similar messages
Oct 19 23:07:03 oak-io5-s2 kernel: Lustre: oak-OST010f: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e760ac3dc00, cur 1603174023 expire 1603173873 last 1603173796
Oct 19 23:07:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 23:09:06 oak-io5-s2 kernel: LustreError: 64880:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00ec: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 1249280 GRANT, real grant 0
Oct 19 23:09:06 oak-io5-s2 kernel: LustreError: 64880:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 5 previous similar messages
Oct 19 23:23:16 oak-io5-s2 kernel: Lustre: oak-OST010a: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 23:23:16 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 19 23:27:03 oak-io5-s2 kernel: Lustre: oak-OST010c: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e62b53ec800, cur 1603175223 expire 1603175073 last 1603174996
Oct 19 23:27:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 19 23:43:16 oak-io5-s2 kernel: Lustre: oak-OST010c: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 19 23:43:16 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 19 23:47:03 oak-io5-s2 kernel: Lustre: oak-OST00cf: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e76143e1000, cur 1603176423 expire 1603176273 last 1603176196
Oct 19 23:47:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 00:03:16 oak-io5-s2 kernel: Lustre: oak-OST00ed: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 00:03:16 oak-io5-s2 kernel: Lustre: Skipped 74 previous similar messages
Oct 20 00:07:03 oak-io5-s2 kernel: Lustre: oak-OST0101: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7b2a671800, cur 1603177623 expire 1603177473 last 1603177396
Oct 20 00:07:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 00:15:46 oak-io5-s2 kernel: LustreError: 94042:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00d4: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 4218880 GRANT, real grant 2576384
Oct 20 00:18:12 oak-io5-s2 kernel: LustreError: 64579:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00ee: cli 50dab6dc-c0b5-4 claims 4218880 GRANT, real grant 1290240
Oct 20 00:18:12 oak-io5-s2 kernel: LustreError: 64579:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 41 previous similar messages
Oct 20 00:19:37 oak-io5-s2 kernel: LustreError: 107339:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 561152 GRANT, real grant 0
Oct 20 00:19:37 oak-io5-s2 kernel: LustreError: 107339:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 2 previous similar messages
Oct 20 00:23:16 oak-io5-s2 kernel: Lustre: oak-OST0103: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 00:23:16 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 00:24:14 oak-io5-s2 kernel: LustreError: 107345:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 258048 GRANT, real grant 0
Oct 20 00:24:14 oak-io5-s2 kernel: LustreError: 107345:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 1 previous similar message
Oct 20 00:27:03 oak-io5-s2 kernel: Lustre: oak-OST010f: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6ce08c9400, cur 1603178823 expire 1603178673 last 1603178596
Oct 20 00:27:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 00:29:10 oak-io5-s2 kernel: LustreError: 64887:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00ec: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 4218880 GRANT, real grant 0
Oct 20 00:29:10 oak-io5-s2 kernel: LustreError: 64887:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 3 previous similar messages
Oct 20 00:40:37 oak-io5-s2 kernel: LustreError: 63813:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 4218880 GRANT, real grant 0
Oct 20 00:40:37 oak-io5-s2 kernel: LustreError: 63813:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 2 previous similar messages
Oct 20 00:43:16 oak-io5-s2 kernel: Lustre: oak-OST010c: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 00:43:16 oak-io5-s2 kernel: Lustre: Skipped 77 previous similar messages
Oct 20 00:47:03 oak-io5-s2 kernel: Lustre: oak-OST00cf: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6cd92ca800, cur 1603180023 expire 1603179873 last 1603179796
Oct 20 00:47:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 01:03:16 oak-io5-s2 kernel: Lustre: oak-OST010c: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 01:03:16 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 01:07:03 oak-io5-s2 kernel: Lustre: oak-OST010f: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e78f8652800, cur 1603181223 expire 1603181073 last 1603180996
Oct 20 01:07:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 01:23:16 oak-io5-s2 kernel: Lustre: oak-OST0104: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 01:23:16 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 20 01:27:03 oak-io5-s2 kernel: Lustre: oak-OST0103: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7a4e83a400, cur 1603182423 expire 1603182273 last 1603182196
Oct 20 01:27:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 01:30:26 oak-io5-s2 kernel: LustreError: 347789:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 258048 GRANT, real grant 0
Oct 20 01:30:26 oak-io5-s2 kernel: LustreError: 347789:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 32 previous similar messages
Oct 20 01:43:15 oak-io5-s2 kernel: Lustre: oak-OST0104: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 01:43:15 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 20 01:47:02 oak-io5-s2 kernel: Lustre: oak-OST00ed: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7412c14400, cur 1603183622 expire 1603183472 last 1603183395
Oct 20 01:47:02 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 02:01:11 oak-io5-s2 kernel: LustreError: 107345:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 73728 GRANT, real grant 0
Oct 20 02:03:15 oak-io5-s2 kernel: Lustre: oak-OST0104: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 02:03:15 oak-io5-s2 kernel: Lustre: Skipped 76 previous similar messages
Oct 20 02:07:02 oak-io5-s2 kernel: Lustre: oak-OST0104: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e77c5b38c00, cur 1603184822 expire 1603184672 last 1603184595
Oct 20 02:07:02 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 02:23:15 oak-io5-s2 kernel: Lustre: oak-OST010f: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 02:23:15 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 20 02:27:02 oak-io5-s2 kernel: Lustre: oak-OST00ed: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7617810800, cur 1603186022 expire 1603185872 last 1603185795
Oct 20 02:27:02 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 02:43:16 oak-io5-s2 kernel: Lustre: oak-OST010a: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 02:43:16 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 20 02:47:03 oak-io5-s2 kernel: Lustre: oak-OST0103: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7761dedc00, cur 1603187223 expire 1603187073 last 1603186996
Oct 20 02:47:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 02:59:40 oak-io5-s2 kernel: LustreError: 347837:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 258048 GRANT, real grant 0
Oct 20 03:03:16 oak-io5-s2 kernel: Lustre: oak-OST00cf: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 03:03:16 oak-io5-s2 kernel: Lustre: Skipped 76 previous similar messages
Oct 20 03:07:03 oak-io5-s2 kernel: Lustre: oak-OST0103: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7413af7c00, cur 1603188423 expire 1603188273 last 1603188196
Oct 20 03:07:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 03:16:41 oak-io5-s2 kernel: LustreError: 347832:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 73728 GRANT, real grant 0
Oct 20 03:19:52 oak-io5-s2 kernel: LustreError: 347790:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00ed: cli af146c4d-8b0e-699a-c653-e467ecdb4220 claims 4218880 GRANT, real grant 618496
Oct 20 03:23:16 oak-io5-s2 kernel: Lustre: oak-OST0103: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 03:23:16 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 03:24:53 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds
Oct 20 03:24:53 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Skipped 5 previous similar messages
Oct 20 03:24:53 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.0.2.250@o2ib5 (4): c: 6, oc: 0, rc: 7
Oct 20 03:24:53 oak-io5-s2 kernel: LNetError: 28355:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Skipped 5 previous similar messages
Oct 20 03:24:53 oak-io5-s2 kernel: LustreError: 28355:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8e87b357a000
Oct 20 03:24:59 oak-io5-s2 kernel: LustreError: 64887:0:(ldlm_lib.c:3285:target_bulk_io()) @@@ network error on bulk READ  req@ffff8e9a6f4b3850 x1673049610335824/t0(0) o3->83583fcb-e285-b0ae-4a2d-e2af30e147d3@10.0.2.250@o2ib5:625/0 lens 608/440 e 1 to 0 dl 1603189520 ref 1 fl Interpret:/0/0 rc 0/0
Oct 20 03:24:59 oak-io5-s2 kernel: Lustre: oak-OST00ed: Bulk IO read error with 83583fcb-e285-b0ae-4a2d-e2af30e147d3 (at 10.0.2.250@o2ib5), client will retry: rc -110
Oct 20 03:25:03 oak-io5-s2 kernel: Lustre: oak-OST00ed: Client 83583fcb-e285-b0ae-4a2d-e2af30e147d3 (at 10.0.2.250@o2ib5) reconnecting
Oct 20 03:25:03 oak-io5-s2 kernel: Lustre: Skipped 2 previous similar messages
Oct 20 03:25:28 oak-io5-s2 kernel: LustreError: 347825:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 258048 GRANT, real grant 0
Oct 20 03:25:28 oak-io5-s2 kernel: LustreError: 347825:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 1 previous similar message
Oct 20 03:27:03 oak-io5-s2 kernel: Lustre: oak-OST00cf: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7a02a59800, cur 1603189623 expire 1603189473 last 1603189396
Oct 20 03:27:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 03:43:16 oak-io5-s2 kernel: Lustre: oak-OST010c: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 03:43:16 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 03:47:03 oak-io5-s2 kernel: Lustre: oak-OST00cf: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6d62e6f800, cur 1603190823 expire 1603190673 last 1603190596
Oct 20 03:47:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 03:59:19 oak-io5-s2 kernel: LustreError: 347852:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 73728 GRANT, real grant 0
Oct 20 04:03:16 oak-io5-s2 kernel: Lustre: oak-OST0101: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 04:03:16 oak-io5-s2 kernel: Lustre: Skipped 77 previous similar messages
Oct 20 04:07:03 oak-io5-s2 kernel: Lustre: oak-OST00e9: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e74139f6c00, cur 1603192023 expire 1603191873 last 1603191796
Oct 20 04:07:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 04:18:20 oak-io5-s2 kernel: LustreError: 347867:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00d7: cli 99c173c4-9d09-4 claims 4206592 GRANT, real grant 4194304
Oct 20 04:19:27 oak-io5-s2 kernel: LustreError: 347805:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 73728 GRANT, real grant 0
Oct 20 04:23:15 oak-io5-s2 kernel: Lustre: oak-OST0103: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 04:23:15 oak-io5-s2 kernel: Lustre: Skipped 80 previous similar messages
Oct 20 04:27:02 oak-io5-s2 kernel: Lustre: oak-OST00ed: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6cc53e6800, cur 1603193222 expire 1603193072 last 1603192995
Oct 20 04:27:02 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 04:43:16 oak-io5-s2 kernel: Lustre: oak-OST0103: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 04:43:16 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 20 04:43:47 oak-io5-s2 kernel: LustreError: 347830:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 1249280 GRANT, real grant 0
Oct 20 04:45:12 oak-io5-s2 kernel: LustreError: 347897:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 1249280 GRANT, real grant 0
Oct 20 04:47:03 oak-io5-s2 kernel: Lustre: oak-OST010a: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6ccbd7f800, cur 1603194423 expire 1603194273 last 1603194196
Oct 20 04:47:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 05:03:15 oak-io5-s2 kernel: Lustre: oak-OST010f: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 05:03:15 oak-io5-s2 kernel: Lustre: Skipped 77 previous similar messages
Oct 20 05:07:02 oak-io5-s2 kernel: Lustre: oak-OST00c0: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6cbacef400, cur 1603195622 expire 1603195472 last 1603195395
Oct 20 05:07:02 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 05:07:40 oak-io5-s2 kernel: LustreError: 347889:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 1609728 GRANT, real grant 0
Oct 20 05:23:16 oak-io5-s2 kernel: Lustre: oak-OST00c0: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 05:23:16 oak-io5-s2 kernel: Lustre: oak-OST0101: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 05:23:16 oak-io5-s2 kernel: Lustre: Skipped 70 previous similar messages
Oct 20 05:23:16 oak-io5-s2 kernel: Lustre: Skipped 8 previous similar messages
Oct 20 05:27:03 oak-io5-s2 kernel: Lustre: oak-OST00c0: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6cc8b97c00, cur 1603196823 expire 1603196673 last 1603196596
Oct 20 05:27:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 05:41:28 oak-io5-s2 kernel: LustreError: 63799:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 1249280 GRANT, real grant 0
Oct 20 05:43:15 oak-io5-s2 kernel: Lustre: oak-OST00c0: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 05:43:15 oak-io5-s2 kernel: Lustre: oak-OST010c: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 05:43:15 oak-io5-s2 kernel: Lustre: Skipped 71 previous similar messages
Oct 20 05:43:15 oak-io5-s2 kernel: Lustre: Skipped 7 previous similar messages
Oct 20 05:47:02 oak-io5-s2 kernel: Lustre: oak-OST00ed: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e7a1e625000, cur 1603198022 expire 1603197872 last 1603197795
Oct 20 05:47:02 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 05:55:09 oak-io5-s2 kernel: LustreError: 347821:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 73728 GRANT, real grant 0
Oct 20 06:03:15 oak-io5-s2 kernel: Lustre: oak-OST00cf: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 06:03:15 oak-io5-s2 kernel: Lustre: Skipped 78 previous similar messages
Oct 20 06:07:02 oak-io5-s2 kernel: Lustre: oak-OST00ed: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6ea6a63c00, cur 1603199222 expire 1603199072 last 1603198995
Oct 20 06:07:02 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 06:11:23 oak-io5-s2 kernel: LustreError: 65289:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST00f2: cli 0899ede3-4b99-0b7d-cb91-8036f430c72b claims 1249280 GRANT, real grant 0
Oct 20 06:13:19 oak-io5-s2 kernel: Lustre: Failing over oak-OST0108
Oct 20 06:13:19 oak-io5-s2 kernel: Lustre: Skipped 6 previous similar messages
Oct 20 06:13:19 oak-io5-s2 kernel: Lustre: oak-OST00dc: Not available for connect from 10.50.9.54@o2ib2 (stopping)
Oct 20 06:13:19 oak-io5-s2 kernel: Lustre: Skipped 7 previous similar messages
Oct 20 06:13:19 oak-io5-s2 kernel: LustreError: 159678:0:(ldlm_resource.c:1147:ldlm_resource_complain()) filter-oak-OST00dc_UUID: namespace resource [0x100abc2:0x0:0x0].0x0 (ffff8e78c12580c0) refcount nonzero (2) after lock cleanup; forcing cleanup.
Oct 20 06:13:19 oak-io5-s2 kernel: LustreError: 159678:0:(ldlm_resource.c:1147:ldlm_resource_complain()) Skipped 32 previous similar messages
Oct 20 06:13:20 oak-io5-s2 kernel: LustreError: 20102:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.210.12.113@tcp1 arrived at 1603199600 with bad export cookie 1088641394522321858
Oct 20 06:13:20 oak-io5-s2 kernel: LustreError: 20102:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 22 previous similar messages
Oct 20 06:13:21 oak-io5-s2 kernel: LustreError: 404:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.210.12.9@tcp1 arrived at 1603199601 with bad export cookie 1088641394522302937
Oct 20 06:13:21 oak-io5-s2 kernel: LustreError: 404:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 38 previous similar messages
Oct 20 06:13:21 oak-io5-s2 kernel: LustreError: 137-5: oak-OST00f4_UUID: not available for connect from 10.49.22.8@o2ib1 (no target). If you are running an HA pair check that the target is mounted on the other server.
Oct 20 06:13:21 oak-io5-s2 kernel: LustreError: Skipped 22021 previous similar messages
Oct 20 06:13:21 oak-io5-s2 kernel: Lustre: server umount oak-OST00f4 complete
Oct 20 06:13:23 oak-io5-s2 kernel: md52: detected capacity change from 112003075014656 to 0
Oct 20 06:13:23 oak-io5-s2 kernel: md: md52 stopped.
Oct 20 06:13:24 oak-io5-s2 kernel: LustreError: 25669:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.50.0.71@o2ib2 arrived at 1603199604 with bad export cookie 1088641394522215983
Oct 20 06:13:24 oak-io5-s2 kernel: LustreError: 25669:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 39 previous similar messages
Oct 20 06:13:26 oak-io5-s2 kernel: LustreError: 25669:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.50.14.13@o2ib2 arrived at 1603199606 with bad export cookie 1088641394523585834
Oct 20 06:13:26 oak-io5-s2 kernel: LustreError: 25669:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 13 previous similar messages
Oct 20 06:13:30 oak-io5-s2 kernel: LustreError: 71736:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.210.12.113@tcp1 arrived at 1603199610 with bad export cookie 1088641394522258564
Oct 20 06:13:30 oak-io5-s2 kernel: LustreError: 71736:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 6 previous similar messages
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST010c is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 10. Is it stuck?
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdbffa7000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdc2c76800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8e760c363c00 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdd35fec00 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8e6cc82aa000 447d4ade-8cde-4 10.51.1.53@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8edb089c5800 089e971d-d416-4 10.51.1.37@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00c4: UNLINKED ffff8ecdf7b96800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00c4: UNLINKED ffff8ed0d234ac00 ca18fa70-e252-4 10.51.1.44@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ecdd3c2cc00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed75311b000 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed724944000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed9318f3c00 b3ae2efd-47c4-4 10.51.1.34@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed23ab1f400 5f8df1bb-2246-4 10.51.1.39@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8e740b73e400 ab5bbcfd-2667-4 10.51.1.50@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecde8e72800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdde3a9c00 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdf16e5c00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdf7772000 ba2aebd3-ab1e-4 10.51.13.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecf1773a000 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ecdf3e48c00 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ece0633cc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ecde5eb7800 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecddcc0fc00 797170a1-7062-4 10.51.13.23@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecdebb94800 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecdde3a5800 2c0c269e-20cf-4 10.51.2.7@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ece057ce000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ecdde261000 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ed938e70400 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ecde03fa000 ba2aebd3-ab1e-4 10.51.13.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecdde3a9400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecddc08fc00 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecde5eb3800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ecdd22b4c00 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ecdf6ebfc00 9dc6008f-964b-4 10.51.2.12@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ed239b8d800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8e6ccd738800 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8ecdf627b000 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8ecde6790400 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST010a: UNLINKED ffff8ecdd5811400 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST010a: UNLINKED ffff8ecde6791800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdc2c70000 a295cbd9-2f2f-4 10.51.2.22@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdcb9c3400 a7eea721-2d42-4 10.51.1.24@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdd8a73400 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ece4fa44400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ed21faedc00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ecdd9164400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ecdf7f59000 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ed9238e0c00 e9172eba-f192-4 10.51.2.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8e705865ac00 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ece4fa40800 089e971d-d416-4 10.51.1.37@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ecdd8e65000 7157ec2d-f8ca-4 10.51.12.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ecdfa684400 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8ece00635000 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8e6cc9584000 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8e6cc2d24400 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdd22b1000 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8ed239b8dc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdebb95000 5fb25fa9-baa3-4 10.51.12.11@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8e6cc66fb000 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdf627fc00 ca18fa70-e252-4 10.51.1.44@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ece08350c00 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d2: UNLINKED ffff8ece08350000 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST0100: UNLINKED ffff8e77b529cc00 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST0100: UNLINKED ffff8ecdd5c40c00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00f6: UNLINKED ffff8ecdd0652800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00f6: UNLINKED ffff8ece0775e400 5fb25fa9-baa3-4 10.51.12.11@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecddf37bc00 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5e0d000 920ea455-6743-4 10.51.1.7@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5eb2400 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5eb0800 e6114cf8-5b4c-4 10.51.2.26@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde57eac00 5685d399-2785-4 10.51.2.6@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ecdce30ec00 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ecdf7b91000 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ece0467f800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ece08356000 1c3239ac-d52e-4 10.51.2.3@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8e6cc66ff000 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ed739d3e800 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8edb089c0800 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00c0: UNLINKED ffff8e752b914800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00c0: UNLINKED ffff8ecdd9167400 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00de: UNLINKED ffff8ecddcc0a800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00de: UNLINKED ffff8ecdde87c400 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ecde7982800 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ed0d234e400 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ecdde879800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ca: UNLINKED ffff8ecdd748d400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ca: UNLINKED ffff8ece4c36fc00 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00fa: UNLINKED ffff8ecdc8a5a800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdcf85dc00 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdffb30800 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdeabb3800 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecde1037c00 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ece0467ec00 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdf7f5bc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdff805400 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdca85c000 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdd35ff400 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdebb90c00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdde3a5c00 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdc2c75400 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ed214fc8000 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdce318800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdce318000 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ece0467a400 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdf7777400 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ece0775d800 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdffecec00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecde6797000 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecde6797c00 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ece0567c800 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8e6cc2907800 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8e6cc96e2800 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ecdf7f5c400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ece08355800 60d506e4-66c0-4 10.51.1.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00fc: UNLINKED ffff8ecdffb37c00 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00fc: UNLINKED ffff8ed938e71800 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ece00635800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ecdeabb7400 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ecdf627c000 e6114cf8-5b4c-4 10.51.2.26@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ece04c86c00 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00f0: UNLINKED ffff8ece06bd6c00 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdf7b95400 a295cbd9-2f2f-4 10.51.2.22@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdc8a5d800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecddef61800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdde3a3c00 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ecdd9162c00 5a52275f-a5b6-4 10.51.1.38@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ecdf0ea6c00 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8e6cc82ad000 c894f8a9-30d7-4 10.51.1.59@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ece08355c00 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ed0d2349400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecde8e77800 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecde8e77400 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecf1773d400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00f2: UNLINKED ffff8ecdf3e4e800 a7eea721-2d42-4 10.51.1.24@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00f2: UNLINKED ffff8ecdd3c2a000 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ea: UNLINKED ffff8ecdca85e800 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST00ea: UNLINKED ffff8ecde5d73400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: Skipped 38 previous similar messages
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ecdc2ecc800 797170a1-7062-4 10.51.13.23@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ecddf37d400 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8edafb86bc00 ab5bbcfd-2667-4 10.51.1.50@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:34 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ed71416b400 79a83f3f-8d8d-4 10.51.1.42@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:38 oak-io5-s2 kernel: LustreError: 68785:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) ldlm_cancel from 10.50.13.10@o2ib2 arrived at 1603199618 with bad export cookie 1088641394522135448
Oct 20 06:13:38 oak-io5-s2 kernel: LustreError: 68785:0:(ldlm_lockd.c:2324:ldlm_cancel_handler()) Skipped 26 previous similar messages
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST010e is waiting for obd_unlinked_exports more than 16 seconds. The obd refcount = 14. Is it stuck?
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c4: UNLINKED ffff8ecdf7b96800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c4: UNLINKED ffff8ed0d234ac00 ca18fa70-e252-4 10.51.1.44@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecde8e72800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdde3a9c00 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdf16e5c00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdf7772000 ba2aebd3-ab1e-4 10.51.13.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecf1773a000 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ecdd3c2cc00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed75311b000 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed724944000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed9318f3c00 b3ae2efd-47c4-4 10.51.1.34@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed23ab1f400 5f8df1bb-2246-4 10.51.1.39@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8e740b73e400 ab5bbcfd-2667-4 10.51.1.50@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ecdf3e48c00 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ece0633cc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ecde5eb7800 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecddcc0fc00 797170a1-7062-4 10.51.13.23@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecdebb94800 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecdde3a5800 2c0c269e-20cf-4 10.51.2.7@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ece057ce000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ecdde261000 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ed938e70400 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ecde03fa000 ba2aebd3-ab1e-4 10.51.13.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecdde3a9400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecddc08fc00 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecde5eb3800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ecdd22b4c00 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ecdf6ebfc00 9dc6008f-964b-4 10.51.2.12@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ed239b8d800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8e6ccd738800 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8ecdf627b000 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8ecde6790400 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST010a: UNLINKED ffff8ecdd5811400 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST010a: UNLINKED ffff8ecde6791800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdc2c70000 a295cbd9-2f2f-4 10.51.2.22@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdcb9c3400 a7eea721-2d42-4 10.51.1.24@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdd8a73400 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ece4fa44400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ed21faedc00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ecdd9164400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ecdf7f59000 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ed9238e0c00 e9172eba-f192-4 10.51.2.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8e705865ac00 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ece4fa40800 089e971d-d416-4 10.51.1.37@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8ece00635000 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ecdd8e65000 7157ec2d-f8ca-4 10.51.12.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8e6cc2d24400 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ecdfa684400 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8ed239b8dc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8e6cc66fb000 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8e6cc9584000 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdd22b1000 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdebb95000 5fb25fa9-baa3-4 10.51.12.11@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdf627fc00 ca18fa70-e252-4 10.51.1.44@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ece08350c00 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d2: UNLINKED ffff8ece08350000 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST0100: UNLINKED ffff8e77b529cc00 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST0100: UNLINKED ffff8ecdd5c40c00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00f6: UNLINKED ffff8ecdd0652800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00f6: UNLINKED ffff8ece0775e400 5fb25fa9-baa3-4 10.51.12.11@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecddf37bc00 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5e0d000 920ea455-6743-4 10.51.1.7@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5eb2400 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5eb0800 e6114cf8-5b4c-4 10.51.2.26@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde57eac00 5685d399-2785-4 10.51.2.6@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ecdce30ec00 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ecdf7b91000 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ece0467f800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ece08356000 1c3239ac-d52e-4 10.51.2.3@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8e6cc66ff000 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ed739d3e800 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8edb089c0800 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00de: UNLINKED ffff8ecddcc0a800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00de: UNLINKED ffff8ecdde87c400 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c0: UNLINKED ffff8e752b914800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00c0: UNLINKED ffff8ecdd9167400 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ecde7982800 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ed0d234e400 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ecdde879800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ca: UNLINKED ffff8ecdd748d400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ca: UNLINKED ffff8ece4c36fc00 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00fa: UNLINKED ffff8ecdc8a5a800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdcf85dc00 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdffb30800 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdff805400 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdeabb3800 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdca85c000 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecde1037c00 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdd35ff400 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ece0467ec00 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdebb90c00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdf7f5bc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdde3a5c00 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdc2c75400 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ed214fc8000 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdce318800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdce318000 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ece0467a400 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdf7777400 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ece0775d800 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdffecec00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecde6797000 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecde6797c00 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ece0567c800 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8e6cc2907800 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8e6cc96e2800 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ecdf7f5c400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ece08355800 60d506e4-66c0-4 10.51.1.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ece00635800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ecdeabb7400 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00fc: UNLINKED ffff8ecdffb37c00 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ecdf627c000 e6114cf8-5b4c-4 10.51.2.26@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00fc: UNLINKED ffff8ed938e71800 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ece04c86c00 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00f0: UNLINKED ffff8ece06bd6c00 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdf7b95400 a295cbd9-2f2f-4 10.51.2.22@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdc8a5d800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecddef61800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdde3a3c00 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ecdd9162c00 5a52275f-a5b6-4 10.51.1.38@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ecdf0ea6c00 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8e6cc82ad000 c894f8a9-30d7-4 10.51.1.59@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ece08355c00 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ed0d2349400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecde8e77800 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecde8e77400 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecf1773d400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00f2: UNLINKED ffff8ecdf3e4e800 a7eea721-2d42-4 10.51.1.24@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00f2: UNLINKED ffff8ecdd3c2a000 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ea: UNLINKED ffff8ecdca85e800 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST00ea: UNLINKED ffff8ecde5d73400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: Skipped 37 previous similar messages
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdbffa7000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdc2c76800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8e760c363c00 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdd35fec00 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8e6cc82aa000 447d4ade-8cde-4 10.51.1.53@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8edb089c5800 089e971d-d416-4 10.51.1.37@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST010c is waiting for obd_unlinked_exports more than 16 seconds. The obd refcount = 10. Is it stuck?
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ecdc2ecc800 797170a1-7062-4 10.51.13.23@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:50 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ecddf37d400 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:51 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8edafb86bc00 ab5bbcfd-2667-4 10.51.1.50@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:13:51 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ed71416b400 79a83f3f-8d8d-4 10.51.1.42@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c4 is waiting for obd_unlinked_exports more than 32 seconds. The obd refcount = 6. Is it stuck?
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecde8e72800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdde3a9c00 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdf16e5c00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdf7772000 ba2aebd3-ab1e-4 10.51.13.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecf1773a000 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ecdd3c2cc00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed75311b000 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed724944000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed9318f3c00 b3ae2efd-47c4-4 10.51.1.34@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed23ab1f400 5f8df1bb-2246-4 10.51.1.39@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8e740b73e400 ab5bbcfd-2667-4 10.51.1.50@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ecdf3e48c00 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ece0633cc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ecde5eb7800 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecddcc0fc00 797170a1-7062-4 10.51.13.23@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecdebb94800 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecdde3a5800 2c0c269e-20cf-4 10.51.2.7@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ece057ce000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ecdde261000 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ed938e70400 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ecde03fa000 ba2aebd3-ab1e-4 10.51.13.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecdde3a9400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecddc08fc00 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecde5eb3800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ecdd22b4c00 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ecdf6ebfc00 9dc6008f-964b-4 10.51.2.12@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ed239b8d800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8e6ccd738800 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8ecdf627b000 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8ecde6790400 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST010a: UNLINKED ffff8ecdd5811400 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST010a: UNLINKED ffff8ecde6791800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdc2c70000 a295cbd9-2f2f-4 10.51.2.22@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdcb9c3400 a7eea721-2d42-4 10.51.1.24@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdd8a73400 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ece4fa44400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ed21faedc00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ecdd9164400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ecdf7f59000 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ed9238e0c00 e9172eba-f192-4 10.51.2.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8e705865ac00 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8e6cc9584000 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdd22b1000 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdebb95000 5fb25fa9-baa3-4 10.51.12.11@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdf627fc00 ca18fa70-e252-4 10.51.1.44@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ece08350c00 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8ece00635000 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8e6cc2d24400 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8ed239b8dc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8e6cc66fb000 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ece4fa40800 089e971d-d416-4 10.51.1.37@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ecdd8e65000 7157ec2d-f8ca-4 10.51.12.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ecdfa684400 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d2: UNLINKED ffff8ece08350000 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00f6: UNLINKED ffff8ecdd0652800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00f6: UNLINKED ffff8ece0775e400 5fb25fa9-baa3-4 10.51.12.11@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST0100: UNLINKED ffff8e77b529cc00 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST0100: UNLINKED ffff8ecdd5c40c00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecddf37bc00 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5e0d000 920ea455-6743-4 10.51.1.7@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5eb2400 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5eb0800 e6114cf8-5b4c-4 10.51.2.26@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde57eac00 5685d399-2785-4 10.51.2.6@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ecdce30ec00 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ecdf7b91000 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ece0467f800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ece08356000 1c3239ac-d52e-4 10.51.2.3@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8e6cc66ff000 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ed739d3e800 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8edb089c0800 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00de: UNLINKED ffff8ecddcc0a800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00de: UNLINKED ffff8ecdde87c400 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c0: UNLINKED ffff8e752b914800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c0: UNLINKED ffff8ecdd9167400 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ecde7982800 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ed0d234e400 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ecdde879800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ca: UNLINKED ffff8ecdd748d400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ca: UNLINKED ffff8ece4c36fc00 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00fa: UNLINKED ffff8ecdc8a5a800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdff805400 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdca85c000 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdcf85dc00 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdd35ff400 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdffb30800 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdebb90c00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdeabb3800 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdde3a5c00 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecde1037c00 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ece0467ec00 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdf7f5bc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdc2c75400 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ed214fc8000 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdce318800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdce318000 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ece0467a400 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdf7777400 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ece0775d800 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdffecec00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecde6797000 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecde6797c00 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ece0567c800 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8e6cc2907800 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8e6cc96e2800 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ecdf7f5c400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ece08355800 60d506e4-66c0-4 10.51.1.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ece00635800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ecdeabb7400 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ecdf627c000 e6114cf8-5b4c-4 10.51.2.26@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ece04c86c00 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00fc: UNLINKED ffff8ecdffb37c00 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00fc: UNLINKED ffff8ed938e71800 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00f0: UNLINKED ffff8ece06bd6c00 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdf7b95400 a295cbd9-2f2f-4 10.51.2.22@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdc8a5d800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecddef61800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdde3a3c00 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ecdd9162c00 5a52275f-a5b6-4 10.51.1.38@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ecdf0ea6c00 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8e6cc82ad000 c894f8a9-30d7-4 10.51.1.59@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ece08355c00 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ed0d2349400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecde8e77800 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecde8e77400 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecf1773d400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00f2: UNLINKED ffff8ecdf3e4e800 a7eea721-2d42-4 10.51.1.24@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00f2: UNLINKED ffff8ecdd3c2a000 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ea: UNLINKED ffff8ecdca85e800 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00ea: UNLINKED ffff8ecde5d73400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: LustreError: 165087:0:(qsd_reint.c:56:qsd_reint_completion()) oak-OST00c4: failed to enqueue global quota lock, glb fid:[0x200000006:0x1020000:0x0], rc:-108
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: Skipped 36 previous similar messages
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c4: UNLINKED ffff8ecdf7b96800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST00c4: UNLINKED ffff8ed0d234ac00 ca18fa70-e252-4 10.51.1.44@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdbffa7000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdc2c76800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8e760c363c00 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdd35fec00 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8e6cc82aa000 447d4ade-8cde-4 10.51.1.53@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:22 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8edb089c5800 089e971d-d416-4 10.51.1.37@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:23 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ecdc2ecc800 797170a1-7062-4 10.51.13.23@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:23 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ecddf37d400 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:23 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8edafb86bc00 ab5bbcfd-2667-4 10.51.1.50@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:23 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ed71416b400 79a83f3f-8d8d-4 10.51.1.42@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:14:29 oak-io5-s2 kernel: LustreError: 165659:0:(qsd_reint.c:56:qsd_reint_completion()) oak-OST00ee: failed to enqueue global quota lock, glb fid:[0x200000006:0x1020000:0x0], rc:-108
Oct 20 06:14:31 oak-io5-s2 kernel: LustreError: 165802:0:(qsd_reint.c:56:qsd_reint_completion()) oak-OST00c2: failed to enqueue global quota lock, glb fid:[0x200000006:0x1020000:0x0], rc:-108
Oct 20 06:14:34 oak-io5-s2 kernel: Lustre: oak-OST00c2: Not available for connect from 10.50.2.18@o2ib2 (stopping)
Oct 20 06:14:34 oak-io5-s2 kernel: Lustre: Skipped 123331 previous similar messages
Oct 20 06:14:36 oak-io5-s2 kernel: LustreError: 166145:0:(qsd_reint.c:56:qsd_reint_completion()) oak-OST0106: failed to enqueue global quota lock, glb fid:[0x200000006:0x1020000:0x0], rc:-108
Oct 20 06:14:36 oak-io5-s2 kernel: LustreError: 166145:0:(qsd_reint.c:56:qsd_reint_completion()) Skipped 1 previous similar message
Oct 20 06:14:36 oak-io5-s2 kernel: LustreError: 137-5: oak-OST00f4_UUID: not available for connect from 10.50.13.11@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
Oct 20 06:14:36 oak-io5-s2 kernel: LustreError: Skipped 3030 previous similar messages
Oct 20 06:14:40 oak-io5-s2 kernel: LustreError: 166517:0:(qsd_reint.c:56:qsd_reint_completion()) oak-OST00c8: failed to enqueue global quota lock, glb fid:[0x200000006:0x1020000:0x0], rc:-108
Oct 20 06:14:40 oak-io5-s2 kernel: LustreError: 166517:0:(qsd_reint.c:56:qsd_reint_completion()) Skipped 3 previous similar messages
Oct 20 06:14:48 oak-io5-s2 kernel: LustreError: 167242:0:(qsd_reint.c:56:qsd_reint_completion()) oak-OST00ea: failed to enqueue global quota lock, glb fid:[0x200000006:0x1020000:0x0], rc:-108
Oct 20 06:14:48 oak-io5-s2 kernel: LustreError: 167242:0:(qsd_reint.c:56:qsd_reint_completion()) Skipped 10 previous similar messages
Oct 20 06:15:06 oak-io5-s2 kernel: LustreError: 168627:0:(qsd_reint.c:56:qsd_reint_completion()) oak-OST0100: failed to enqueue global quota lock, glb fid:[0x200000006:0x1020000:0x0], rc:-108
Oct 20 06:15:06 oak-io5-s2 kernel: LustreError: 168627:0:(qsd_reint.c:56:qsd_reint_completion()) Skipped 13 previous similar messages
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ec is waiting for obd_unlinked_exports more than 64 seconds. The obd refcount = 12. Is it stuck?
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ecdd3c2cc00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed75311b000 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed724944000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed9318f3c00 b3ae2efd-47c4-4 10.51.1.34@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed23ab1f400 5f8df1bb-2246-4 10.51.1.39@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8e740b73e400 ab5bbcfd-2667-4 10.51.1.50@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ecdf3e48c00 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ece0633cc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ecde5eb7800 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecddcc0fc00 797170a1-7062-4 10.51.13.23@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecdebb94800 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecdde3a5800 2c0c269e-20cf-4 10.51.2.7@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ece057ce000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ecdde261000 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ed938e70400 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ecde03fa000 ba2aebd3-ab1e-4 10.51.13.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecdde3a9400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecddc08fc00 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecde5eb3800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ecdd22b4c00 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ecdf6ebfc00 9dc6008f-964b-4 10.51.2.12@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ed239b8d800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8e6ccd738800 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8ecdf627b000 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8ecde6790400 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST010a: UNLINKED ffff8ecdd5811400 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST010a: UNLINKED ffff8ecde6791800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdc2c70000 a295cbd9-2f2f-4 10.51.2.22@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdcb9c3400 a7eea721-2d42-4 10.51.1.24@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdd8a73400 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ece4fa44400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ed21faedc00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ecdd9164400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ecdf7f59000 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ed9238e0c00 e9172eba-f192-4 10.51.2.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8e705865ac00 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ece4fa40800 089e971d-d416-4 10.51.1.37@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ecdd8e65000 7157ec2d-f8ca-4 10.51.12.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ecdfa684400 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8ece00635000 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8e6cc2d24400 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8ed239b8dc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8e6cc66fb000 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8e6cc9584000 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdd22b1000 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdebb95000 5fb25fa9-baa3-4 10.51.12.11@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdf627fc00 ca18fa70-e252-4 10.51.1.44@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ece08350c00 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d2: UNLINKED ffff8ece08350000 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST0100: UNLINKED ffff8e77b529cc00 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST0100: UNLINKED ffff8ecdd5c40c00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00f6: UNLINKED ffff8ecdd0652800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00f6: UNLINKED ffff8ece0775e400 5fb25fa9-baa3-4 10.51.12.11@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecddf37bc00 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5e0d000 920ea455-6743-4 10.51.1.7@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5eb2400 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ecdce30ec00 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5eb0800 e6114cf8-5b4c-4 10.51.2.26@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde57eac00 5685d399-2785-4 10.51.2.6@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ecdf7b91000 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ece0467f800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ece08356000 1c3239ac-d52e-4 10.51.2.3@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8e6cc66ff000 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ed739d3e800 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8edb089c0800 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c0: UNLINKED ffff8e752b914800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c0: UNLINKED ffff8ecdd9167400 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ecde7982800 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ed0d234e400 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ecdde879800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00de: UNLINKED ffff8ecddcc0a800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00de: UNLINKED ffff8ecdde87c400 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ca: UNLINKED ffff8ecdd748d400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ca: UNLINKED ffff8ece4c36fc00 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00fa: UNLINKED ffff8ecdc8a5a800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdff805400 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdca85c000 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdd35ff400 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdebb90c00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdde3a5c00 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdcf85dc00 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdffb30800 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdeabb3800 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecde1037c00 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ece0467ec00 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdf7f5bc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdc2c75400 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ed214fc8000 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdce318800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdce318000 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ece0467a400 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdf7777400 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ece0775d800 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdffecec00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecde6797000 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecde6797c00 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ece0567c800 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8e6cc2907800 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8e6cc96e2800 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ecdf7f5c400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ece08355800 60d506e4-66c0-4 10.51.1.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ece00635800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ecdeabb7400 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ecdf627c000 e6114cf8-5b4c-4 10.51.2.26@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ece04c86c00 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00fc: UNLINKED ffff8ecdffb37c00 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00fc: UNLINKED ffff8ed938e71800 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00f0: UNLINKED ffff8ece06bd6c00 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdf7b95400 a295cbd9-2f2f-4 10.51.2.22@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdc8a5d800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecddef61800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdde3a3c00 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ecdd9162c00 5a52275f-a5b6-4 10.51.1.38@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ecdf0ea6c00 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8e6cc82ad000 c894f8a9-30d7-4 10.51.1.59@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ece08355c00 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ed0d2349400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecde8e77800 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecde8e77400 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecf1773d400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00f2: UNLINKED ffff8ecdf3e4e800 a7eea721-2d42-4 10.51.1.24@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00f2: UNLINKED ffff8ecdd3c2a000 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ea: UNLINKED ffff8ecdca85e800 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ea: UNLINKED ffff8ecde5d73400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: Skipped 37 previous similar messages
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecde8e72800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdde3a9c00 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdf16e5c00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdf7772000 ba2aebd3-ab1e-4 10.51.13.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecf1773a000 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c4: UNLINKED ffff8ecdf7b96800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:26 oak-io5-s2 kernel: Lustre: oak-OST00c4: UNLINKED ffff8ed0d234ac00 ca18fa70-e252-4 10.51.1.44@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:27 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdbffa7000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:27 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdc2c76800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:27 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8e760c363c00 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:27 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdd35fec00 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:27 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8e6cc82aa000 447d4ade-8cde-4 10.51.1.53@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:27 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8edb089c5800 089e971d-d416-4 10.51.1.37@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:27 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ecdc2ecc800 797170a1-7062-4 10.51.13.23@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:27 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ecddf37d400 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:27 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8edafb86bc00 ab5bbcfd-2667-4 10.51.1.50@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:15:27 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ed71416b400 79a83f3f-8d8d-4 10.51.1.42@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:16:22 oak-io5-s2 kernel: LustreError: 174890:0:(qsd_reint.c:56:qsd_reint_completion()) oak-OST00c4: failed to enqueue global quota lock, glb fid:[0x200000006:0x1020000:0x0], rc:-108
Oct 20 06:16:22 oak-io5-s2 kernel: LustreError: 174890:0:(qsd_reint.c:56:qsd_reint_completion()) Skipped 5 previous similar messages
Oct 20 06:17:04 oak-io5-s2 kernel: Lustre: oak-OST00c0: Not available for connect from 10.51.1.23@o2ib3 (stopping)
Oct 20 06:17:04 oak-io5-s2 kernel: Lustre: Skipped 184785 previous similar messages
Oct 20 06:17:06 oak-io5-s2 kernel: LustreError: 137-5: oak-OST00f4_UUID: not available for connect from 10.50.15.9@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
Oct 20 06:17:06 oak-io5-s2 kernel: LustreError: Skipped 4783 previous similar messages
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST0108 is waiting for obd_unlinked_exports more than 128 seconds. The obd refcount = 8. Is it stuck?
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ecdd3c2cc00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed75311b000 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed724944000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed9318f3c00 b3ae2efd-47c4-4 10.51.1.34@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed23ab1f400 5f8df1bb-2246-4 10.51.1.39@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8e740b73e400 ab5bbcfd-2667-4 10.51.1.50@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecddcc0fc00 797170a1-7062-4 10.51.13.23@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecdebb94800 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecdde3a5800 2c0c269e-20cf-4 10.51.2.7@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ece057ce000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ecdde261000 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ed938e70400 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ecde03fa000 ba2aebd3-ab1e-4 10.51.13.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecdde3a9400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecddc08fc00 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecde5eb3800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ecdd22b4c00 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ecdf6ebfc00 9dc6008f-964b-4 10.51.2.12@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ed239b8d800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8e6ccd738800 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8ecdf627b000 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8ecde6790400 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST010a: UNLINKED ffff8ecdd5811400 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST010a: UNLINKED ffff8ecde6791800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdc2c70000 a295cbd9-2f2f-4 10.51.2.22@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdcb9c3400 a7eea721-2d42-4 10.51.1.24@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdd8a73400 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ece4fa44400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ed21faedc00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ecdd9164400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ecdf7f59000 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ed9238e0c00 e9172eba-f192-4 10.51.2.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8e705865ac00 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ece4fa40800 089e971d-d416-4 10.51.1.37@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ecdd8e65000 7157ec2d-f8ca-4 10.51.12.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ecdfa684400 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8ece00635000 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8e6cc2d24400 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8ed239b8dc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8e6cc66fb000 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8e6cc9584000 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdd22b1000 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdebb95000 5fb25fa9-baa3-4 10.51.12.11@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdf627fc00 ca18fa70-e252-4 10.51.1.44@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ece08350c00 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST0100: UNLINKED ffff8e77b529cc00 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST0100: UNLINKED ffff8ecdd5c40c00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d2: UNLINKED ffff8ece08350000 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00f6: UNLINKED ffff8ecdd0652800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00f6: UNLINKED ffff8ece0775e400 5fb25fa9-baa3-4 10.51.12.11@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ecdce30ec00 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ecdf7b91000 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ece0467f800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ece08356000 1c3239ac-d52e-4 10.51.2.3@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8e6cc66ff000 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ed739d3e800 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8edb089c0800 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecddf37bc00 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5e0d000 920ea455-6743-4 10.51.1.7@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5eb2400 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5eb0800 e6114cf8-5b4c-4 10.51.2.26@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde57eac00 5685d399-2785-4 10.51.2.6@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00c0: UNLINKED ffff8e752b914800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00c0: UNLINKED ffff8ecdd9167400 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ecde7982800 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ed0d234e400 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ecdde879800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00de: UNLINKED ffff8ecddcc0a800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00de: UNLINKED ffff8ecdde87c400 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ca: UNLINKED ffff8ecdd748d400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ca: UNLINKED ffff8ece4c36fc00 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00fa: UNLINKED ffff8ecdc8a5a800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdcf85dc00 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdffb30800 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdeabb3800 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecde1037c00 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ece0467ec00 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdf7f5bc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdff805400 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdca85c000 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdd35ff400 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdebb90c00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdde3a5c00 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ece0567c800 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8e6cc2907800 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8e6cc96e2800 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ecdf7f5c400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ece08355800 60d506e4-66c0-4 10.51.1.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdc2c75400 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ed214fc8000 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdce318800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ece00635800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdce318000 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ecdeabb7400 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ece0467a400 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ecdf627c000 e6114cf8-5b4c-4 10.51.2.26@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdf7777400 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ece04c86c00 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ece0775d800 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdffecec00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecde6797000 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecde6797c00 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00fc: UNLINKED ffff8ecdffb37c00 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00fc: UNLINKED ffff8ed938e71800 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00f0: UNLINKED ffff8ece06bd6c00 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdf7b95400 a295cbd9-2f2f-4 10.51.2.22@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdc8a5d800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecddef61800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdde3a3c00 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ecdd9162c00 5a52275f-a5b6-4 10.51.1.38@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ecdf0ea6c00 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8e6cc82ad000 c894f8a9-30d7-4 10.51.1.59@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ece08355c00 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ed0d2349400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecde8e77800 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecde8e77400 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecf1773d400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00f2: UNLINKED ffff8ecdf3e4e800 a7eea721-2d42-4 10.51.1.24@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00f2: UNLINKED ffff8ecdd3c2a000 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ea: UNLINKED ffff8ecdca85e800 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ea: UNLINKED ffff8ecde5d73400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: Skipped 37 previous similar messages
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ecdf3e48c00 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ece0633cc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ecde5eb7800 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecde8e72800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdde3a9c00 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdf16e5c00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdf7772000 ba2aebd3-ab1e-4 10.51.13.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecf1773a000 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00c4: UNLINKED ffff8ecdf7b96800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:34 oak-io5-s2 kernel: Lustre: oak-OST00c4: UNLINKED ffff8ed0d234ac00 ca18fa70-e252-4 10.51.1.44@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:35 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdbffa7000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:35 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdc2c76800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:35 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8e760c363c00 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:35 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdd35fec00 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:35 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8e6cc82aa000 447d4ade-8cde-4 10.51.1.53@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:35 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8edb089c5800 089e971d-d416-4 10.51.1.37@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:35 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ecdc2ecc800 797170a1-7062-4 10.51.13.23@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:35 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ecddf37d400 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:35 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8edafb86bc00 ab5bbcfd-2667-4 10.51.1.50@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:17:35 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ed71416b400 79a83f3f-8d8d-4 10.51.1.42@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:18:22 oak-io5-s2 kernel: LustreError: 184755:0:(qsd_reint.c:56:qsd_reint_completion()) oak-OST00c4: failed to enqueue global quota lock, glb fid:[0x200000006:0x1020000:0x0], rc:-108
Oct 20 06:18:22 oak-io5-s2 kernel: LustreError: 184755:0:(qsd_reint.c:56:qsd_reint_completion()) Skipped 38 previous similar messages
Oct 20 06:20:31 oak-io5-s2 kernel: LustreError: 195286:0:(qsd_reint.c:56:qsd_reint_completion()) oak-OST00c2: failed to enqueue global quota lock, glb fid:[0x200000006:0x1020000:0x0], rc:-108
Oct 20 06:20:31 oak-io5-s2 kernel: LustreError: 195286:0:(qsd_reint.c:56:qsd_reint_completion()) Skipped 40 previous similar messages
Oct 20 06:21:49 oak-io5-s2 kernel: Lustre: oak-OST00d0 is waiting for obd_unlinked_exports more than 256 seconds. The obd refcount = 14. Is it stuck?
Oct 20 06:21:49 oak-io5-s2 kernel: Lustre: Skipped 4 previous similar messages
Oct 20 06:21:49 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecddcc0fc00 797170a1-7062-4 10.51.13.23@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:49 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecdebb94800 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:49 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecdde3a5800 2c0c269e-20cf-4 10.51.2.7@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:49 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ece057ce000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:49 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ecdde261000 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:49 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ed938e70400 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:49 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ecde03fa000 ba2aebd3-ab1e-4 10.51.13.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecdde3a9400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecddc08fc00 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecde5eb3800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ecdd22b4c00 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ecdf6ebfc00 9dc6008f-964b-4 10.51.2.12@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ed239b8d800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST010a: UNLINKED ffff8ecdd5811400 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8e6ccd738800 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST010a: UNLINKED ffff8ecde6791800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8ecdf627b000 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8ecde6790400 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdc2c70000 a295cbd9-2f2f-4 10.51.2.22@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdcb9c3400 a7eea721-2d42-4 10.51.1.24@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdd8a73400 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ece4fa44400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ed21faedc00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ecdd9164400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ecdf7f59000 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ed9238e0c00 e9172eba-f192-4 10.51.2.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8e705865ac00 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8ece00635000 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8e6cc2d24400 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8ed239b8dc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8e6cc66fb000 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8e6cc9584000 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdd22b1000 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdebb95000 5fb25fa9-baa3-4 10.51.12.11@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdf627fc00 ca18fa70-e252-4 10.51.1.44@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ece08350c00 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST0100: UNLINKED ffff8e77b529cc00 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d2: UNLINKED ffff8ece08350000 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST0100: UNLINKED ffff8ecdd5c40c00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00f6: UNLINKED ffff8ecdd0652800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00f6: UNLINKED ffff8ece0775e400 5fb25fa9-baa3-4 10.51.12.11@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ece4fa40800 089e971d-d416-4 10.51.1.37@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ecdd8e65000 7157ec2d-f8ca-4 10.51.12.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ecdfa684400 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ecdce30ec00 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecddf37bc00 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ecdf7b91000 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ece0467f800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5e0d000 920ea455-6743-4 10.51.1.7@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ece08356000 1c3239ac-d52e-4 10.51.2.3@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5eb2400 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8e6cc66ff000 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5eb0800 e6114cf8-5b4c-4 10.51.2.26@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ed739d3e800 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde57eac00 5685d399-2785-4 10.51.2.6@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8edb089c0800 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00c0: UNLINKED ffff8e752b914800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00c0: UNLINKED ffff8ecdd9167400 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ecde7982800 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ed0d234e400 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ecdde879800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00de: UNLINKED ffff8ecddcc0a800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00de: UNLINKED ffff8ecdde87c400 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ca: UNLINKED ffff8ecdd748d400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ca: UNLINKED ffff8ece4c36fc00 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00fa: UNLINKED ffff8ecdc8a5a800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdff805400 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdca85c000 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdd35ff400 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdebb90c00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdde3a5c00 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ece0567c800 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8e6cc2907800 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8e6cc96e2800 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ecdf7f5c400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ece08355800 60d506e4-66c0-4 10.51.1.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ece00635800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdc2c75400 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ecdeabb7400 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ed214fc8000 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ecdf627c000 e6114cf8-5b4c-4 10.51.2.26@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdce318800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ece04c86c00 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdce318000 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ece0467a400 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdf7777400 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ece0775d800 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdffecec00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecde6797000 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecde6797c00 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00fc: UNLINKED ffff8ecdffb37c00 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00fc: UNLINKED ffff8ed938e71800 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00f0: UNLINKED ffff8ece06bd6c00 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdcf85dc00 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdffb30800 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdeabb3800 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecde1037c00 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ece0467ec00 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdf7f5bc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdf7b95400 a295cbd9-2f2f-4 10.51.2.22@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdc8a5d800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecddef61800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdde3a3c00 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ecdd9162c00 5a52275f-a5b6-4 10.51.1.38@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ecdf0ea6c00 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8e6cc82ad000 c894f8a9-30d7-4 10.51.1.59@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ece08355c00 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ed0d2349400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecde8e77800 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecde8e77400 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecf1773d400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00f2: UNLINKED ffff8ecdf3e4e800 a7eea721-2d42-4 10.51.1.24@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00f2: UNLINKED ffff8ecdd3c2a000 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ea: UNLINKED ffff8ecdca85e800 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ea: UNLINKED ffff8ecde5d73400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ecdd3c2cc00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed75311b000 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed724944000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed9318f3c00 b3ae2efd-47c4-4 10.51.1.34@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed23ab1f400 5f8df1bb-2246-4 10.51.1.39@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8e740b73e400 ab5bbcfd-2667-4 10.51.1.50@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ecdf3e48c00 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ece0633cc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ecde5eb7800 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecde8e72800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdde3a9c00 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdf16e5c00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00c4: UNLINKED ffff8ecdf7b96800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00c4: UNLINKED ffff8ed0d234ac00 ca18fa70-e252-4 10.51.1.44@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdf7772000 ba2aebd3-ab1e-4 10.51.13.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:50 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecf1773a000 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:51 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ecdc2ecc800 797170a1-7062-4 10.51.13.23@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:51 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ecddf37d400 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:51 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8edafb86bc00 ab5bbcfd-2667-4 10.51.1.50@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:51 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ed71416b400 79a83f3f-8d8d-4 10.51.1.42@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:51 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdbffa7000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:51 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdc2c76800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:51 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8e760c363c00 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:51 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdd35fec00 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:51 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8e6cc82aa000 447d4ade-8cde-4 10.51.1.53@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:21:51 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8edb089c5800 089e971d-d416-4 10.51.1.37@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:22:04 oak-io5-s2 kernel: Lustre: oak-OST00c4: Not available for connect from 10.50.9.59@o2ib2 (stopping)
Oct 20 06:22:04 oak-io5-s2 kernel: Lustre: Skipped 247683 previous similar messages
Oct 20 06:22:07 oak-io5-s2 kernel: LustreError: 137-5: oak-OST00f4_UUID: not available for connect from 10.51.1.26@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
Oct 20 06:22:07 oak-io5-s2 kernel: LustreError: Skipped 6428 previous similar messages
Oct 20 06:23:16 oak-io5-s2 kernel: Lustre: oak-OST0101: Connection restored to  (at 10.49.27.12@o2ib1)
Oct 20 06:23:16 oak-io5-s2 kernel: Lustre: Skipped 73 previous similar messages
Oct 20 06:24:47 oak-io5-s2 kernel: LustreError: 216454:0:(qsd_reint.c:56:qsd_reint_completion()) oak-OST0104: failed to enqueue global quota lock, glb fid:[0x200000006:0x1020000:0x0], rc:-108
Oct 20 06:24:47 oak-io5-s2 kernel: LustreError: 216454:0:(qsd_reint.c:56:qsd_reint_completion()) Skipped 93 previous similar messages
Oct 20 06:27:03 oak-io5-s2 kernel: Lustre: oak-OST00cf: haven't heard from client 5f6215b1-b4ca-4 (at 10.49.27.12@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6ccafc3800, cur 1603200423 expire 1603200273 last 1603200196
Oct 20 06:27:03 oak-io5-s2 kernel: Lustre: Skipped 79 previous similar messages
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST0102 is waiting for obd_unlinked_exports more than 512 seconds. The obd refcount = 8. Is it stuck?
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ece057ce000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ecdde261000 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ed938e70400 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00dc: UNLINKED ffff8ecde03fa000 ba2aebd3-ab1e-4 10.51.13.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ecdd22b4c00 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ecdf6ebfc00 9dc6008f-964b-4 10.51.2.12@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00c8: UNLINKED ffff8ed239b8d800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecdde3a9400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecddc08fc00 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST0106: UNLINKED ffff8ecde5eb3800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8e6ccd738800 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8ecdf627b000 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00c6: UNLINKED ffff8ecde6790400 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST010a: UNLINKED ffff8ecdd5811400 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST010a: UNLINKED ffff8ecde6791800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdc2c70000 a295cbd9-2f2f-4 10.51.2.22@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdcb9c3400 a7eea721-2d42-4 10.51.1.24@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ecdd8a73400 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e2: UNLINKED ffff8ece4fa44400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ed21faedc00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ecdd9164400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ecdf7f59000 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8ed9238e0c00 e9172eba-f192-4 10.51.2.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ce: UNLINKED ffff8e705865ac00 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8e6cc9584000 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8ece00635000 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdd22b1000 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8e6cc2d24400 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdebb95000 5fb25fa9-baa3-4 10.51.12.11@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8ed239b8dc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ecdf627fc00 ca18fa70-e252-4 10.51.1.44@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ee: UNLINKED ffff8e6cc66fb000 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00c2: UNLINKED ffff8ece08350c00 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST0100: UNLINKED ffff8e77b529cc00 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST0100: UNLINKED ffff8ecdd5c40c00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d2: UNLINKED ffff8ece08350000 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00f6: UNLINKED ffff8ecdd0652800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00f6: UNLINKED ffff8ece0775e400 5fb25fa9-baa3-4 10.51.12.11@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecddf37bc00 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5e0d000 920ea455-6743-4 10.51.1.7@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5eb2400 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde5eb0800 e6114cf8-5b4c-4 10.51.2.26@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d6: UNLINKED ffff8ecde57eac00 5685d399-2785-4 10.51.2.6@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ecdce30ec00 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ecdf7b91000 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ece0467f800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ece08356000 1c3239ac-d52e-4 10.51.2.3@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8e6cc66ff000 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8ed739d3e800 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00cc: UNLINKED ffff8edb089c0800 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ece4fa40800 089e971d-d416-4 10.51.1.37@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ecdd8e65000 7157ec2d-f8ca-4 10.51.12.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d4: UNLINKED ffff8ecdfa684400 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ecde7982800 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ed0d234e400 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e4: UNLINKED ffff8ecdde879800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00c0: UNLINKED ffff8e752b914800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00c0: UNLINKED ffff8ecdd9167400 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00de: UNLINKED ffff8ecddcc0a800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00de: UNLINKED ffff8ecdde87c400 15820499-6a67-4 10.51.1.45@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ca: UNLINKED ffff8ecdd748d400 1b493823-5cb0-4 10.51.2.16@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ca: UNLINKED ffff8ece4c36fc00 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdff805400 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdca85c000 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdd35ff400 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdebb90c00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00fe: UNLINKED ffff8ecdde3a5c00 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ece0567c800 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8e6cc2907800 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8e6cc96e2800 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ecdf7f5c400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e8: UNLINKED ffff8ece08355800 60d506e4-66c0-4 10.51.1.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00fa: UNLINKED ffff8ecdc8a5a800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ece00635800 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ecdeabb7400 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ecdf627c000 e6114cf8-5b4c-4 10.51.2.26@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00f8: UNLINKED ffff8ece04c86c00 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdc2c75400 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ed214fc8000 7a090af5-09dc-4 10.51.12.4@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdce318800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdce318000 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ece0467a400 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdf7777400 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ece0775d800 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecdffecec00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecde6797000 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d8: UNLINKED ffff8ecde6797c00 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00fc: UNLINKED ffff8ecdffb37c00 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00fc: UNLINKED ffff8ed938e71800 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00f0: UNLINKED ffff8ece06bd6c00 0fd6c3fc-cdea-4 10.51.1.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdf7b95400 a295cbd9-2f2f-4 10.51.2.22@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdc8a5d800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecddef61800 a03a88db-dd9f-4 10.51.1.70@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e0: UNLINKED ffff8ecdde3a3c00 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdcf85dc00 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdffb30800 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdeabb3800 692d080d-0253-4 10.51.2.28@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecde1037c00 081dcf50-bf97-4 10.51.2.32@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ece0467ec00 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00da: UNLINKED ffff8ecdf7f5bc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ecdd9162c00 5a52275f-a5b6-4 10.51.1.38@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ecdf0ea6c00 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8e6cc82ad000 c894f8a9-30d7-4 10.51.1.59@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ece08355c00 aca3900e-0bd3-4 10.51.12.20@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00e6: UNLINKED ffff8ed0d2349400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecde8e77800 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecde8e77400 f83b2e5a-865f-4 10.51.2.31@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST0104: UNLINKED ffff8ecf1773d400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00f2: UNLINKED ffff8ecdf3e4e800 a7eea721-2d42-4 10.51.1.24@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00f2: UNLINKED ffff8ecdd3c2a000 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ea: UNLINKED ffff8ecdca85e800 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ea: UNLINKED ffff8ecde5d73400 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: Skipped 37 previous similar messages
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecddcc0fc00 797170a1-7062-4 10.51.13.23@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecdebb94800 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST0102: UNLINKED ffff8ecdde3a5800 2c0c269e-20cf-4 10.51.2.7@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ecdd3c2cc00 2df1f7fc-7a05-4 10.51.12.19@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed75311b000 2be2f511-7fb2-4 10.51.1.43@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed724944000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed9318f3c00 b3ae2efd-47c4-4 10.51.1.34@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8ed23ab1f400 5f8df1bb-2246-4 10.51.1.39@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00d0: UNLINKED ffff8e740b73e400 ab5bbcfd-2667-4 10.51.1.50@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ecdf3e48c00 809c6883-20df-4 10.51.1.1@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ece0633cc00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST0108: UNLINKED ffff8ecde5eb7800 a21dfbdb-d869-4 10.51.1.27@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00c4: UNLINKED ffff8ecdf7b96800 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00c4: UNLINKED ffff8ed0d234ac00 ca18fa70-e252-4 10.51.1.44@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecde8e72800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdde3a9c00 3785df25-bcfa-4 10.51.1.36@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:22 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdf16e5c00 30ac26b8-a761-4 10.51.1.60@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:23 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecdf7772000 ba2aebd3-ab1e-4 10.51.13.17@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:23 oak-io5-s2 kernel: Lustre: oak-OST00ec: UNLINKED ffff8ecf1773a000 d456b8d9-e32d-4 10.51.2.25@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:23 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ecdc2ecc800 797170a1-7062-4 10.51.13.23@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:23 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ecddf37d400 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:23 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8edafb86bc00 ab5bbcfd-2667-4 10.51.1.50@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:23 oak-io5-s2 kernel: Lustre: oak-OST010c: UNLINKED ffff8ed71416b400 79a83f3f-8d8d-4 10.51.1.42@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:23 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdbffa7000 c6406ab5-3bb0-4 10.51.0.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:23 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdc2c76800 f2e15550-2e62-4 10.51.1.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:23 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8e760c363c00 ab2cb5a8-57a4-4 10.51.2.18@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:23 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8ecdd35fec00 ff37a732-9b0c-4 10.51.1.52@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:23 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8e6cc82aa000 447d4ade-8cde-4 10.51.1.53@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0
Oct 20 06:30:23 oak-io5-s2 kernel: Lustre: oak-OST010e: UNLINKED ffff8edb089c5800 089e971d-d416-4 10.51.1.37@o2ib3 1 (0 0 0) 1 0 1 0:           (null)  0 stale:0