Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.430991] LustreError: 15235:0:(lu_object.h:867:lu_object_attr()) LBUG Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.438726] Pid: 15235, comm: mdt01_226 Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.443231] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.443232] Call Trace: Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.448067] [] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.456111] [] lbug_with_loc+0x47/0xb0 [libcfs] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.463271] [] mdt_attr_get_complex+0x6da/0x770 [mdt] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.471013] [] mdt_reconstruct_create+0xd5/0x340 [mdt] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.478868] [] ? lu_ucred+0x20/0x30 [obdclass] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.485922] [] mdt_reconstruct+0x45/0x120 [mdt] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.493078] [] mdt_reint_internal+0x6bb/0x780 [mdt] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.500608] [] mdt_reint+0x44/0xe0 [mdt] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.507076] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.514642] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.523062] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.530448] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.539345] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.546603] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.554462] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.562285] [] ? __wake_up+0x53/0x70 Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.568392] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.575658] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.582886] [] child_rip+0xa/0x20 Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.588702] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.595976] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.603203] [] ? child_rip+0x0/0x20 Jul 17 16:34:09 atlas-mds3.ccs.ornl.gov kernel: [794253.609169] Jul 17 16:34:20 atlas-mds1.ccs.ornl.gov kernel: [794026.562685] LustreError: 35497:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 198200163895, new: 198200163894 replay: 0. see LU-617. Jul 17 16:34:20 atlas-mds1.ccs.ornl.gov kernel: [794026.580607] LustreError: 35497:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Skipped 3 previous similar messages Jul 17 16:34:21 atlas-mds1.ccs.ornl.gov kernel: [794027.177357] LustreError: 35497:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 198200168281, new: 198200168280 replay: 0. see LU-617. Jul 17 16:34:39 atlas-oss3h2.ccs.ornl.gov kernel: [1914037.349402] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 133@gni3 ns: filter-atlas2-OST0309_UUID lock: ffff880485607240/0x80482b1b82e79d3e lrc: 3/0,0 mode: PW/PW res: [0x20a589:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 nid: 133@gni3 remote: 0xbe32d2ff6950d903 expref: 4 pid: 26852 timeout: 6207992329 lvb_type: 0 Jul 17 16:34:39 atlas-oss3h2.ccs.ornl.gov kernel: [1914037.481731] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 13 previous similar messages Jul 17 16:34:39 atlas-oss4d1.ccs.ornl.gov kernel: [1914030.509947] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 132@gni3 ns: filter-atlas2-OST02a0_UUID lock: ffff880ed84e1900/0x8f1a2421ddd3a925 lrc: 3/0,0 mode: PW/PW res: [0x210a69:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x20 nid: 132@gni3 remote: 0x23ac0f72fc53d382 expref: 4 pid: 29119 timeout: 6207991151 lvb_type: 0 Jul 17 16:34:40 atlas-oss3h2.ccs.ornl.gov kernel: [1914037.631750] LustreError: 54663:0:(ldlm_lockd.c:2380:ldlm_cancel_handler()) ldlm_cancel from 133@gni3 arrived at 1405629279 with bad export cookie 9243685632284528805 Jul 17 16:44:56 atlas-oss4d4.ccs.ornl.gov kernel: [1914671.175239] Lustre: 27385:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88031965a000 x1471904513679024/t0(0) o400->atlas2-MDT0000-lwp-OST0213@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4d4.ccs.ornl.gov kernel: [1914671.175243] Lustre: 27383:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88083b0e1c00 x1471904513679016/t0(0) o400->atlas2-MDT0000-lwp-OST00f3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4d4.ccs.ornl.gov kernel: [1914671.175254] Lustre: atlas2-MDT0000-lwp-OST00f3: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4d4.ccs.ornl.gov kernel: [1914671.422949] Lustre: atlas2-MDT0000-lwp-OST0213: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4d3.ccs.ornl.gov kernel: [1914648.002797] Lustre: 22338:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880aaecb5000 x1471904516978140/t0(0) o400->atlas2-MDT0000-lwp-OST0332@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4d3.ccs.ornl.gov kernel: [1914648.002808] Lustre: atlas2-MDT0000-lwp-OST03c2: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4d3.ccs.ornl.gov kernel: [1914648.154516] Lustre: 22338:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:56 atlas-oss4g5.ccs.ornl.gov kernel: [1914651.653341] Lustre: 26060:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88071897b400 x1471904510456628/t0(0) o400->atlas2-MDT0000-lwp-OST034c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4g5.ccs.ornl.gov kernel: [1914651.653349] Lustre: atlas2-MDT0000-lwp-OST019c: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4g5.ccs.ornl.gov kernel: [1914651.805369] Lustre: 26060:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:56 atlas-oss4g2.ccs.ornl.gov kernel: [1914662.313234] Lustre: 22146:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8804e3be9800 x1471904509228856/t0(0) o400->atlas2-MDT0000-lwp-OST0349@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4g2.ccs.ornl.gov kernel: [1914662.313245] Lustre: atlas2-MDT0000-lwp-OST03d9: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4h4.ccs.ornl.gov kernel: [1914670.953534] Lustre: 25831:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880716c35c00 x1471904512334240/t0(0) o400->atlas2-MDT0000-lwp-OST0353@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4h4.ccs.ornl.gov kernel: [1914670.953542] Lustre: atlas2-MDT0000-lwp-OST02c3: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4h4.ccs.ornl.gov kernel: [1914671.099411] Lustre: 25831:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:56 atlas-oss4h4.ccs.ornl.gov kernel: [1914671.099439] Lustre: 25830:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8805a777a800 x1471904512334232/t0(0) o400->atlas2-MDT0000-lwp-OST0233@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4h4.ccs.ornl.gov kernel: [1914671.099443] Lustre: 25830:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:56 atlas-oss4h4.ccs.ornl.gov kernel: [1914671.099451] Lustre: atlas2-MDT0000-lwp-OST0233: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4h4.ccs.ornl.gov kernel: [1914671.099454] Lustre: Skipped 3 previous similar messages Jul 17 16:44:56 atlas-oss4e4.ccs.ornl.gov kernel: [1914660.245439] Lustre: 22191:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8806dd4dd400 x1471904515374204/t0(0) o400->atlas2-MDT0000-lwp-OST02ab@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4e4.ccs.ornl.gov kernel: [1914660.245456] Lustre: atlas2-MDT0000-lwp-OST006b: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4e4.ccs.ornl.gov kernel: [1914660.398109] Lustre: 22191:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:56 atlas-oss4f2.ccs.ornl.gov kernel: [1914655.129656] Lustre: 25728:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8801d39a2c00 x1471904508910544/t0(0) o400->atlas2-MDT0000-lwp-OST0071@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4f2.ccs.ornl.gov kernel: [1914655.129665] Lustre: atlas2-MDT0000-lwp-OST0221: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4f2.ccs.ornl.gov kernel: [1914655.273662] Lustre: 25728:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:56 atlas-oss4h7.ccs.ornl.gov kernel: [1914659.847135] Lustre: 26110:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880856738400 x1471904512615036/t0(0) o400->atlas2-MDT0000-lwp-OST0356@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4h7.ccs.ornl.gov kernel: [1914659.847214] Lustre: atlas2-MDT0000-lwp-OST0236: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4h7.ccs.ornl.gov kernel: [1914659.993390] Lustre: 26110:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:56 atlas-oss4h6.ccs.ornl.gov kernel: [1914650.369024] Lustre: 25974:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880a652c7c00 x1471904515531956/t0(0) o400->atlas2-MDT0000-lwp-OST0085@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4h6.ccs.ornl.gov kernel: [1914650.369036] Lustre: atlas2-MDT0000-lwp-OST0355: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4h6.ccs.ornl.gov kernel: [1914650.515227] Lustre: 25974:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:56 atlas-oss4d2.ccs.ornl.gov kernel: [1914656.663944] Lustre: 27440:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88042aa4d400 x1471904512220844/t0(0) o400->atlas2-MDT0000-lwp-OST0331@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4d2.ccs.ornl.gov kernel: [1914656.663953] Lustre: atlas2-MDT0000-lwp-OST0181: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4d2.ccs.ornl.gov kernel: [1914656.812697] Lustre: 27440:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:56 atlas-oss4a4.ccs.ornl.gov kernel: [1914653.121793] Lustre: 27351:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88041faec000 x1471904509590980/t0(0) o400->atlas2-MDT0000-lwp-OST03ab@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4a4.ccs.ornl.gov kernel: [1914653.121954] Lustre: atlas2-MDT0000-lwp-OST004b: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4a4.ccs.ornl.gov kernel: [1914653.268808] Lustre: 27351:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:44:56 atlas-oss4a4.ccs.ornl.gov kernel: [1914653.268852] Lustre: 27363:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88011f0fc400 x1471904509590968/t0(0) o400->atlas2-MDT0000-lwp-OST01fb@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4a4.ccs.ornl.gov kernel: [1914653.268857] Lustre: 27363:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:44:56 atlas-oss4a4.ccs.ornl.gov kernel: [1914653.268866] Lustre: atlas2-MDT0000-lwp-OST01fb: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4a4.ccs.ornl.gov kernel: [1914653.268868] Lustre: Skipped 1 previous similar message Jul 17 16:44:56 atlas-oss4i2.ccs.ornl.gov kernel: [1914674.415016] Lustre: 25746:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880831056c00 x1471904514634420/t0(0) o400->atlas2-MDT0000-lwp-OST03e9@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4i2.ccs.ornl.gov kernel: [1914674.415029] Lustre: atlas2-MDT0000-lwp-OST01a9: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4i2.ccs.ornl.gov kernel: [1914674.415032] Lustre: Skipped 1 previous similar message Jul 17 16:44:56 atlas-oss4i2.ccs.ornl.gov kernel: [1914674.572315] Lustre: 25746:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:56 atlas-oss4i7.ccs.ornl.gov kernel: [1914664.849044] Lustre: 25154:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880d3446dc00 x1471904511374988/t0(0) o400->atlas2-MDT0000-lwp-OST02ce@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4i7.ccs.ornl.gov kernel: [1914664.849058] Lustre: atlas2-MDT0000-lwp-OST03ee: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4i7.ccs.ornl.gov kernel: [1914664.995920] Lustre: 25154:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:56 atlas-oss4g5.ccs.ornl.gov kernel: [1914651.835775] Lustre: atlas2-MDT0000-lwp-OST034c: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4f7.ccs.ornl.gov kernel: [1914650.645820] Lustre: 25987:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8807872fbc00 x1471904510454916/t0(0) o400->atlas2-MDT0000-lwp-OST0196@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4f7.ccs.ornl.gov kernel: [1914650.645839] Lustre: atlas2-MDT0000-lwp-OST02b6: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4g2.ccs.ornl.gov kernel: [1914662.461643] Lustre: 22146:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:56 atlas-oss4g2.ccs.ornl.gov kernel: [1914662.492126] Lustre: atlas2-MDT0000-lwp-OST0349: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4b3.ccs.ornl.gov kernel: [1914645.884941] Lustre: 22300:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8804b459a800 x1471904509180400/t0(0) o400->atlas2-MDT0000-lwp-OST0292@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss3h8.ccs.ornl.gov kernel: [1914648.463003] Lustre: 22181:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880749bd3800 x1471904514772700/t0(0) o400->atlas2-MDT0000-lwp-OST030f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4h7.ccs.ornl.gov kernel: [1914660.023784] Lustre: atlas2-MDT0000-lwp-OST0356: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss3h8.ccs.ornl.gov kernel: [1914648.463973] Lustre: atlas2-MDT0000-lwp-OST039f: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss3h8.ccs.ornl.gov kernel: [1914648.613160] Lustre: 22181:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:56 atlas-oss3d6.ccs.ornl.gov kernel: [1914665.982398] Lustre: 25611:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880bc479e000 x1471904511183264/t0(0) o400->atlas2-MDT0000-lwp-OST037d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss3d6.ccs.ornl.gov kernel: [1914665.982407] Lustre: atlas2-MDT0000-lwp-OST00ad: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss3d6.ccs.ornl.gov kernel: [1914666.127211] Lustre: 25611:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:56 atlas-oss4h6.ccs.ornl.gov kernel: [1914650.545773] Lustre: atlas2-MDT0000-lwp-OST0085: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4d2.ccs.ornl.gov kernel: [1914656.843323] Lustre: atlas2-MDT0000-lwp-OST0331: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4f2.ccs.ornl.gov kernel: [1914655.312599] Lustre: atlas2-MDT0000-lwp-OST0071: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4b2.ccs.ornl.gov kernel: [1914674.387503] Lustre: 27464:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880c75df7000 x1471904518025712/t0(0) o400->atlas2-MDT0000-lwp-OST00e1@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4b2.ccs.ornl.gov kernel: [1914674.387526] Lustre: atlas2-MDT0000-lwp-OST03b1: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4b2.ccs.ornl.gov kernel: [1914674.387529] Lustre: Skipped 1 previous similar message Jul 17 16:44:56 atlas-oss4b2.ccs.ornl.gov kernel: [1914674.545137] Lustre: 27464:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15353345 previous similar messages Jul 17 16:44:56 atlas-oss4i7.ccs.ornl.gov kernel: [1914665.026685] Lustre: atlas2-MDT0000-lwp-OST02ce: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4i2.ccs.ornl.gov kernel: [1914674.611230] Lustre: atlas2-MDT0000-lwp-OST03e9: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4f7.ccs.ornl.gov kernel: [1914650.796671] Lustre: 25987:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:56 atlas-oss4f7.ccs.ornl.gov kernel: [1914650.827345] Lustre: atlas2-MDT0000-lwp-OST0196: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4b3.ccs.ornl.gov kernel: [1914645.884983] Lustre: atlas2-MDT0000-lwp-OST00e2: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4b3.ccs.ornl.gov kernel: [1914646.037349] Lustre: 22300:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:56 atlas-oss3e3.ccs.ornl.gov kernel: [1914648.311394] Lustre: 23063:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88070db5c400 x1471904510468320/t0(0) o400->atlas2-MDT0000-lwp-OST0142@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss3e3.ccs.ornl.gov kernel: [1914648.311401] Lustre: atlas2-MDT0000-lwp-OST0022: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss3e3.ccs.ornl.gov kernel: [1914648.458507] Lustre: 23063:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:44:56 atlas-oss3h8.ccs.ornl.gov kernel: [1914648.643920] Lustre: atlas2-MDT0000-lwp-OST030f: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4g5.ccs.ornl.gov kernel: [1914651.886519] Lustre: Skipped 5 previous similar messages Jul 17 16:44:56 atlas-oss3d6.ccs.ornl.gov kernel: [1914666.157822] Lustre: atlas2-MDT0000-lwp-OST037d: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4d7.ccs.ornl.gov kernel: [1914652.327965] Lustre: 22282:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880b4394e000 x1471904514719852/t0(0) o400->atlas2-MDT0000-lwp-OST0186@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4d7.ccs.ornl.gov kernel: [1914652.328000] Lustre: atlas2-MDT0000-lwp-OST0066: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4d7.ccs.ornl.gov kernel: [1914652.328003] Lustre: Skipped 1 previous similar message Jul 17 16:44:56 atlas-oss4d7.ccs.ornl.gov kernel: [1914652.482898] Lustre: 22282:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:56 atlas-oss4b2.ccs.ornl.gov kernel: [1914674.584442] Lustre: atlas2-MDT0000-lwp-OST00e1: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4e2.ccs.ornl.gov kernel: [1914665.418597] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880c5d7abc00 x1471904514146216/t0(0) o400->atlas2-MDT0000-lwp-OST02a9@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4e2.ccs.ornl.gov kernel: [1914665.418654] Lustre: atlas2-MDT0000-lwp-OST0219: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4e2.ccs.ornl.gov kernel: [1914665.418657] Lustre: Skipped 1 previous similar message Jul 17 16:44:56 atlas-oss4e2.ccs.ornl.gov kernel: [1914665.588089] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:56 atlas-oss4h7.ccs.ornl.gov kernel: [1914660.074678] Lustre: Skipped 5 previous similar messages Jul 17 16:44:56 atlas-oss3h5.ccs.ornl.gov kernel: [1914671.349381] Lustre: 24607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880363788800 x1471904507806472/t0(0) o400->atlas2-MDT0000-lwp-OST039c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4e4.ccs.ornl.gov kernel: [1914660.398194] Lustre: atlas2-MDT0000-lwp-OST018b: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss3e3.ccs.ornl.gov kernel: [1914648.488970] Lustre: atlas2-MDT0000-lwp-OST0142: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4f2.ccs.ornl.gov kernel: [1914655.363523] Lustre: Skipped 4 previous similar messages Jul 17 16:44:56 atlas-oss3f6.ccs.ornl.gov kernel: [801356.240228] Lustre: 79674:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880ab6001000 x1473076856558444/t0(0) o400->atlas2-MDT0000-lwp-OST002d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss3f6.ccs.ornl.gov kernel: [801356.240361] Lustre: atlas2-MDT0000-lwp-OST00bd: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss3f6.ccs.ornl.gov kernel: [801356.385390] Lustre: 79674:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:56 atlas-oss4h6.ccs.ornl.gov kernel: [1914650.605019] Lustre: Skipped 5 previous similar messages Jul 17 16:44:56 atlas-oss4d2.ccs.ornl.gov kernel: [1914656.902760] Lustre: Skipped 5 previous similar messages Jul 17 16:44:56 atlas-oss4e1.ccs.ornl.gov kernel: [1914649.524818] Lustre: 26046:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880cca04c000 x1471904515144620/t0(0) o400->atlas2-MDT0000-lwp-OST0188@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4e1.ccs.ornl.gov kernel: [1914649.524838] Lustre: atlas2-MDT0000-lwp-OST03c8: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4e1.ccs.ornl.gov kernel: [1914649.675440] Lustre: 26046:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:56 atlas-oss4g2.ccs.ornl.gov kernel: [1914662.551460] Lustre: Skipped 5 previous similar messages Jul 17 16:44:56 atlas-oss4i2.ccs.ornl.gov kernel: [1914674.662289] Lustre: Skipped 4 previous similar messages Jul 17 16:44:56 atlas-oss4b3.ccs.ornl.gov kernel: [1914646.067731] Lustre: atlas2-MDT0000-lwp-OST0292: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss3i7.ccs.ornl.gov kernel: [1914648.711924] Lustre: 24586:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880bdf62e400 x1471904508389160/t0(0) o400->atlas2-MDT0000-lwp-OST00d6@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4h2.ccs.ornl.gov kernel: [1914672.806761] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88077aa86c00 x1471904508411000/t0(0) o400->atlas2-MDT0000-lwp-OST0081@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss3i7.ccs.ornl.gov kernel: [1914648.712890] Lustre: 24587:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880a71a73000 x1471904508389172/t0(0) o400->atlas2-MDT0000-lwp-OST0286@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss3i7.ccs.ornl.gov kernel: [1914648.712901] Lustre: atlas2-MDT0000-lwp-OST0286: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss3i7.ccs.ornl.gov kernel: [1914648.962439] Lustre: 24586:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:56 atlas-oss4h2.ccs.ornl.gov kernel: [1914672.806773] Lustre: atlas2-MDT0000-lwp-OST01a1: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4h2.ccs.ornl.gov kernel: [1914672.808761] Lustre: atlas2-MDT0000-lwp-OST03e1: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4h2.ccs.ornl.gov kernel: [1914672.808764] Lustre: Skipped 3 previous similar messages Jul 17 16:44:56 atlas-oss4h2.ccs.ornl.gov kernel: [1914673.026500] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:56 atlas-oss4i7.ccs.ornl.gov kernel: [1914665.085870] Lustre: Skipped 5 previous similar messages Jul 17 16:44:56 atlas-oss4f2.ccs.ornl.gov kernel: [1914655.383331] Lustre: 25728:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88021299ac00 x1471904508910548/t0(0) o400->atlas2-MDT0000-lwp-OST0101@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4c5.ccs.ornl.gov kernel: [1914647.332601] Lustre: 26660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8804bbda8c00 x1471904510171100/t0(0) o400->atlas2-MDT0000-lwp-OST029c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4c5.ccs.ornl.gov kernel: [1914647.333614] Lustre: 26658:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88076d812800 x1471904510171088/t0(0) o400->atlas2-MDT0000-lwp-OST00ec@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4c5.ccs.ornl.gov kernel: [1914647.333625] Lustre: atlas2-MDT0000-lwp-OST00ec: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4c5.ccs.ornl.gov kernel: [1914647.571490] Lustre: 26660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:56 atlas-oss4c5.ccs.ornl.gov kernel: [1914647.571522] Lustre: atlas2-MDT0000-lwp-OST005c: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4c5.ccs.ornl.gov kernel: [1914647.571526] Lustre: Skipped 1 previous similar message Jul 17 16:44:56 atlas-oss4d7.ccs.ornl.gov kernel: [1914652.521731] Lustre: atlas2-MDT0000-lwp-OST0186: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4c3.ccs.ornl.gov kernel: [1914648.464244] Lustre: 24967:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880d85de8000 x1471904512245940/t0(0) o400->atlas2-MDT0000-lwp-OST00ea@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4c3.ccs.ornl.gov kernel: [1914648.464281] Lustre: atlas2-MDT0000-lwp-OST029a: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss3h7.ccs.ornl.gov kernel: [1914658.389884] Lustre: 25086:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88073df55c00 x1471904510332812/t0(0) o400->atlas2-MDT0000-lwp-OST01ee@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss3h7.ccs.ornl.gov kernel: [1914658.389896] Lustre: atlas2-MDT0000-lwp-OST003e: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss3h7.ccs.ornl.gov kernel: [1914658.539429] Lustre: 25086:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:56 atlas-oss3h5.ccs.ornl.gov kernel: [1914671.349392] Lustre: atlas2-MDT0000-lwp-OST00cc: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss3h5.ccs.ornl.gov kernel: [1914671.495552] Lustre: 24607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:56 atlas-oss4b3.ccs.ornl.gov kernel: [1914646.118813] Lustre: Skipped 5 previous similar messages Jul 17 16:44:56 atlas-oss3h8.ccs.ornl.gov kernel: [1914648.703170] Lustre: Skipped 4 previous similar messages Jul 17 16:44:56 atlas-oss3f6.ccs.ornl.gov kernel: [801356.415776] Lustre: atlas2-MDT0000-lwp-OST002d: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss3d6.ccs.ornl.gov kernel: [1914666.217016] Lustre: Skipped 4 previous similar messages Jul 17 16:44:56 atlas-oss4c1.ccs.ornl.gov kernel: [1914657.180372] Lustre: 22145:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88065e678c00 x1471904516097380/t0(0) o400->atlas2-MDT0000-lwp-OST0298@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss4c1.ccs.ornl.gov kernel: [1914657.180420] Lustre: atlas2-MDT0000-lwp-OST0328: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4c1.ccs.ornl.gov kernel: [1914657.330519] Lustre: 22145:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:56 atlas-oss4e1.ccs.ornl.gov kernel: [1914649.705920] Lustre: atlas2-MDT0000-lwp-OST0188: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss4f7.ccs.ornl.gov kernel: [1914650.886800] Lustre: Skipped 5 previous similar messages Jul 17 16:44:56 atlas-oss4b2.ccs.ornl.gov kernel: [1914674.635282] Lustre: Skipped 4 previous similar messages Jul 17 16:44:56 atlas-oss3g8.ccs.ornl.gov kernel: [1913249.740030] Lustre: 13445:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880e4e790400 x1471904513602284/t0(0) o400->atlas2-MDT0000-lwp-OST0157@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:56 atlas-oss3i7.ccs.ornl.gov kernel: [1914648.993008] Lustre: atlas2-MDT0000-lwp-OST00d6: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss3g8.ccs.ornl.gov kernel: [1913249.740059] Lustre: atlas2-MDT0000-lwp-OST0277: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:56 atlas-oss3g8.ccs.ornl.gov kernel: [1913249.890332] Lustre: 13445:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3h8.ccs.ornl.gov kernel: [1914648.714621] Lustre: 22181:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880800797800 x1471904514772696/t0(0) o400->atlas2-MDT0000-lwp-OST027f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4b1.ccs.ornl.gov kernel: [1914656.193769] Lustre: 25872:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880427449800 x1471904507948952/t0(0) o400->atlas2-MDT0000-lwp-OST00e0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3d6.ccs.ornl.gov kernel: [1914666.228006] Lustre: 25611:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880c65b43000 x1471904511183260/t0(0) o400->atlas2-MDT0000-lwp-OST02ed@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4b1.ccs.ornl.gov kernel: [1914656.193784] Lustre: atlas2-MDT0000-lwp-OST0200: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4b1.ccs.ornl.gov kernel: [1914656.338470] Lustre: 25872:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3e3.ccs.ornl.gov kernel: [1914648.539976] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4i3.ccs.ornl.gov kernel: [1914643.871044] Lustre: 25965:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88034ad54000 x1471904523159456/t0(0) o400->atlas2-MDT0000-lwp-OST023a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4i3.ccs.ornl.gov kernel: [1914643.871076] Lustre: atlas2-MDT0000-lwp-OST011a: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4i3.ccs.ornl.gov kernel: [1914643.876052] Lustre: atlas2-MDT0000-lwp-OST03ea: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4i3.ccs.ornl.gov kernel: [1914643.876056] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4c3.ccs.ornl.gov kernel: [1914648.609178] Lustre: 24967:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss4c3.ccs.ornl.gov kernel: [1914648.639837] Lustre: atlas2-MDT0000-lwp-OST00ea: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3h7.ccs.ornl.gov kernel: [1914658.569873] Lustre: atlas2-MDT0000-lwp-OST01ee: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3f5.ccs.ornl.gov kernel: [801343.622034] Lustre: 79205:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880b4e355000 x1473076856406168/t0(0) o400->atlas2-MDT0000-lwp-OST026c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3f5.ccs.ornl.gov kernel: [801343.622116] Lustre: atlas2-MDT0000-lwp-OST02fc: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3f5.ccs.ornl.gov kernel: [801343.765993] Lustre: 79205:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss4i4.ccs.ornl.gov kernel: [1914656.432365] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff881065fc3800 x1471904516513292/t0(0) o400->atlas2-MDT0000-lwp-OST008b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4i4.ccs.ornl.gov kernel: [1914656.432409] Lustre: atlas2-MDT0000-lwp-OST01ab: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4i4.ccs.ornl.gov kernel: [1914656.439350] Lustre: 22170:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880cd3714400 x1471904516513304/t0(0) o400->atlas2-MDT0000-lwp-OST023b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4i4.ccs.ornl.gov kernel: [1914656.439355] Lustre: 22170:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4i4.ccs.ornl.gov kernel: [1914656.439364] Lustre: atlas2-MDT0000-lwp-OST023b: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4c1.ccs.ornl.gov kernel: [1914657.360881] Lustre: atlas2-MDT0000-lwp-OST0298: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g8.ccs.ornl.gov kernel: [1913249.920868] Lustre: atlas2-MDT0000-lwp-OST0157: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3c1.ccs.ornl.gov kernel: [1914653.779507] Lustre: 23691:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8802d4d72400 x1471904511392832/t0(0) o400->atlas2-MDT0000-lwp-OST0010@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3c1.ccs.ornl.gov kernel: [1914653.779515] Lustre: atlas2-MDT0000-lwp-OST0250: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3c1.ccs.ornl.gov kernel: [1914653.796501] Lustre: atlas2-MDT0000-lwp-OST0130: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3c1.ccs.ornl.gov kernel: [1914653.796504] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3c1.ccs.ornl.gov kernel: [1914654.001568] Lustre: 23691:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3f8.ccs.ornl.gov kernel: [801354.111559] Lustre: 81087:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88064ea92400 x1473076861320816/t0(0) o400->atlas2-MDT0000-lwp-OST02ff@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3f8.ccs.ornl.gov kernel: [801354.111612] Lustre: atlas2-MDT0000-lwp-OST014f: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3f8.ccs.ornl.gov kernel: [801354.111615] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4d7.ccs.ornl.gov kernel: [1914652.572359] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4b1.ccs.ornl.gov kernel: [1914656.369022] Lustre: atlas2-MDT0000-lwp-OST00e0: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g5.ccs.ornl.gov kernel: [1913253.312602] Lustre: 13565:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88018a999c00 x1471904518248944/t0(0) o400->atlas2-MDT0000-lwp-OST0034@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3h5.ccs.ornl.gov kernel: [1914671.525828] Lustre: atlas2-MDT0000-lwp-OST039c: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g5.ccs.ornl.gov kernel: [1913253.312616] Lustre: atlas2-MDT0000-lwp-OST00c4: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g5.ccs.ornl.gov kernel: [1913253.461523] Lustre: 13565:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss4f6.ccs.ornl.gov kernel: [1914646.497875] Lustre: 22286:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88097d89d000 x1471904511708416/t0(0) o400->atlas2-MDT0000-lwp-OST0075@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4f6.ccs.ornl.gov kernel: [1914646.497905] Lustre: atlas2-MDT0000-lwp-OST0345: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4f6.ccs.ornl.gov kernel: [1914646.497909] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4f6.ccs.ornl.gov kernel: [1914646.654375] Lustre: 22286:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3a6.ccs.ornl.gov kernel: [1914654.366285] Lustre: 25782:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88031b3ef400 x1471904512394508/t0(0) o400->atlas2-MDT0000-lwp-OST0125@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3a6.ccs.ornl.gov kernel: [1914654.366302] Lustre: atlas2-MDT0000-lwp-OST0005: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3a6.ccs.ornl.gov kernel: [1914654.512899] Lustre: 25782:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3f6.ccs.ornl.gov kernel: [801356.466617] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4c2.ccs.ornl.gov kernel: [1914655.674894] Lustre: 25987:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880063e71c00 x1471904508570528/t0(0) o400->atlas2-MDT0000-lwp-OST00e9@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4c2.ccs.ornl.gov kernel: [1914655.674933] Lustre: atlas2-MDT0000-lwp-OST0179: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4c2.ccs.ornl.gov kernel: [1914655.819166] Lustre: 25987:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss4i3.ccs.ornl.gov kernel: [1914644.088938] Lustre: 25965:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3i7.ccs.ornl.gov kernel: [1914649.044002] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4c4.ccs.ornl.gov kernel: [1914663.532749] Lustre: 25795:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880cb8874800 x1471904510980940/t0(0) o400->atlas2-MDT0000-lwp-OST032b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4c4.ccs.ornl.gov kernel: [1914663.532836] Lustre: atlas2-MDT0000-lwp-OST03bb: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4c4.ccs.ornl.gov kernel: [1914663.542725] Lustre: atlas2-MDT0000-lwp-OST020b: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4c4.ccs.ornl.gov kernel: [1914663.542728] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4c4.ccs.ornl.gov kernel: [1914663.753702] Lustre: 25795:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5769484 previous similar messages Jul 17 16:44:57 atlas-oss4e1.ccs.ornl.gov kernel: [1914649.765498] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3h5.ccs.ornl.gov kernel: [1914671.576805] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4g6.ccs.ornl.gov kernel: [1914661.423481] Lustre: 24695:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8806c0c9ec00 x1471904507898564/t0(0) o400->atlas2-MDT0000-lwp-OST03dd@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3f5.ccs.ornl.gov kernel: [801343.804693] Lustre: atlas2-MDT0000-lwp-OST026c: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4i4.ccs.ornl.gov kernel: [1914656.765985] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3f8.ccs.ornl.gov kernel: [801354.275228] Lustre: 81087:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3f8.ccs.ornl.gov kernel: [801354.305517] Lustre: atlas2-MDT0000-lwp-OST02ff: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3h7.ccs.ornl.gov kernel: [1914658.620719] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3f4.ccs.ornl.gov kernel: [1914653.860048] Lustre: 22156:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880b6c737000 x1471904507778360/t0(0) o400->atlas2-MDT0000-lwp-OST038b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e7.ccs.ornl.gov kernel: [1914654.160856] Lustre: 25890:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880cc9916c00 x1471904508195068/t0(0) o400->atlas2-MDT0000-lwp-OST0026@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3f4.ccs.ornl.gov kernel: [1914653.860068] Lustre: atlas2-MDT0000-lwp-OST00bb: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3f4.ccs.ornl.gov kernel: [1914653.862037] Lustre: atlas2-MDT0000-lwp-OST014b: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3f4.ccs.ornl.gov kernel: [1914653.862040] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3f4.ccs.ornl.gov kernel: [1914654.080097] Lustre: 22156:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5020308 previous similar messages Jul 17 16:44:57 atlas-oss3e7.ccs.ornl.gov kernel: [1914654.160974] Lustre: atlas2-MDT0000-lwp-OST0146: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e7.ccs.ornl.gov kernel: [1914654.304794] Lustre: 25890:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3e7.ccs.ornl.gov kernel: [1914654.304827] Lustre: 25891:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880e73677c00 x1471904508195072/t0(0) o400->atlas2-MDT0000-lwp-OST00b6@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e7.ccs.ornl.gov kernel: [1914654.304832] Lustre: 25891:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3e7.ccs.ornl.gov kernel: [1914654.304841] Lustre: atlas2-MDT0000-lwp-OST00b6: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e7.ccs.ornl.gov kernel: [1914654.304843] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3g5.ccs.ornl.gov kernel: [1913253.492284] Lustre: atlas2-MDT0000-lwp-OST0034: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3a6.ccs.ornl.gov kernel: [1914654.543257] Lustre: atlas2-MDT0000-lwp-OST0125: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3a1.ccs.ornl.gov kernel: [1914646.598593] Lustre: 22442:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880ff0b48000 x1471904525931628/t0(0) o400->atlas2-MDT0000-lwp-OST0120@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3b5.ccs.ornl.gov kernel: [1914664.771949] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880416cf9400 x1471904514716844/t0(0) o400->atlas2-MDT0000-lwp-OST02dc@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3b5.ccs.ornl.gov kernel: [1914664.771965] Lustre: atlas2-MDT0000-lwp-OST01bc: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b5.ccs.ornl.gov kernel: [1914664.920802] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4e1.ccs.ornl.gov kernel: [1914649.785139] Lustre: 26046:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880be8e63000 x1471904515144616/t0(0) o400->atlas2-MDT0000-lwp-OST00f8@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4c1.ccs.ornl.gov kernel: [1914657.411712] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4c2.ccs.ornl.gov kernel: [1914655.849592] Lustre: atlas2-MDT0000-lwp-OST00e9: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4d3.ccs.ornl.gov kernel: [1914648.154595] Lustre: 22340:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880f86fda000 x1471904516978144/t0(0) o400->atlas2-MDT0000-lwp-OST02a2@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4d3.ccs.ornl.gov kernel: [1914648.154599] Lustre: 22340:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4d3.ccs.ornl.gov kernel: [1914648.154606] Lustre: atlas2-MDT0000-lwp-OST02a2: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4d3.ccs.ornl.gov kernel: [1914648.154608] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4b6.ccs.ornl.gov kernel: [1914649.399906] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8801535e7000 x1471904511213428/t0(0) o400->atlas2-MDT0000-lwp-OST00e5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3h7.ccs.ornl.gov kernel: [1914658.640609] Lustre: 25086:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88060cba7400 x1471904510332808/t0(0) o400->atlas2-MDT0000-lwp-OST015e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4b6.ccs.ornl.gov kernel: [1914649.399917] Lustre: atlas2-MDT0000-lwp-OST0295: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4b6.ccs.ornl.gov kernel: [1914649.548051] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss4g7.ccs.ornl.gov kernel: [1914674.073180] Lustre: 26066:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff881029f7fc00 x1471904508873448/t0(0) o400->atlas2-MDT0000-lwp-OST022e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4g7.ccs.ornl.gov kernel: [1914674.073213] Lustre: atlas2-MDT0000-lwp-OST03de: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4g7.ccs.ornl.gov kernel: [1914674.220059] Lustre: 26066:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3h2.ccs.ornl.gov kernel: [1914654.727350] Lustre: 20919:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880ffd4c8c00 x1471904518760364/t0(0) o400->atlas2-MDT0000-lwp-OST0159@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3h2.ccs.ornl.gov kernel: [1914654.727356] Lustre: 20923:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff881071f93c00 x1471904518760376/t0(0) o400->atlas2-MDT0000-lwp-OST0309@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3h2.ccs.ornl.gov kernel: [1914654.727368] Lustre: atlas2-MDT0000-lwp-OST0309: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3h2.ccs.ornl.gov kernel: [1914654.973746] Lustre: 20919:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4h8.ccs.ornl.gov kernel: [1914642.726796] Lustre: 24633:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8801e047cc00 x1471904514267084/t0(0) o400->atlas2-MDT0000-lwp-OST03e7@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4h8.ccs.ornl.gov kernel: [1914642.726837] Lustre: atlas2-MDT0000-lwp-OST02c7: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g8.ccs.ornl.gov kernel: [1913249.979972] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3h4.ccs.ornl.gov kernel: [1914655.563929] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880e1166d400 x1471904509245128/t0(0) o400->atlas2-MDT0000-lwp-OST015b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3h4.ccs.ornl.gov kernel: [1914655.563940] Lustre: atlas2-MDT0000-lwp-OST039b: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3h4.ccs.ornl.gov kernel: [1914655.712515] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4c3.ccs.ornl.gov kernel: [1914648.699118] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3e1.ccs.ornl.gov kernel: [1914654.317786] Lustre: 23611:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880624e32000 x1471904509694204/t0(0) o400->atlas2-MDT0000-lwp-OST0020@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e1.ccs.ornl.gov kernel: [1914654.317804] Lustre: atlas2-MDT0000-lwp-OST0260: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e1.ccs.ornl.gov kernel: [1914654.317808] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3e1.ccs.ornl.gov kernel: [1914654.319771] Lustre: 23616:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88027d24fc00 x1471904509694224/t0(0) o400->atlas2-MDT0000-lwp-OST0380@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e1.ccs.ornl.gov kernel: [1914654.319775] Lustre: 23616:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3e1.ccs.ornl.gov kernel: [1914654.319784] Lustre: atlas2-MDT0000-lwp-OST0380: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e1.ccs.ornl.gov kernel: [1914654.319786] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3e1.ccs.ornl.gov kernel: [1914654.682928] Lustre: 23611:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4b1.ccs.ornl.gov kernel: [1914656.428521] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4c8.ccs.ornl.gov kernel: [1914643.457932] Lustre: 22055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880380008800 x1471904512196796/t0(0) o400->atlas2-MDT0000-lwp-OST032f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4c8.ccs.ornl.gov kernel: [1914643.457942] Lustre: atlas2-MDT0000-lwp-OST017f: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4c8.ccs.ornl.gov kernel: [1914643.458924] Lustre: 22053:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88054fd85c00 x1471904512196792/t0(0) o400->atlas2-MDT0000-lwp-OST029f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4c8.ccs.ornl.gov kernel: [1914643.458928] Lustre: 22053:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4c8.ccs.ornl.gov kernel: [1914643.458934] Lustre: atlas2-MDT0000-lwp-OST005f: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4c8.ccs.ornl.gov kernel: [1914643.458940] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss4c8.ccs.ornl.gov kernel: [1914643.803110] Lustre: 22055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4e7.ccs.ornl.gov kernel: [1914649.003098] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8803e80ef400 x1471904522621388/t0(0) o400->atlas2-MDT0000-lwp-OST02ae@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4e7.ccs.ornl.gov kernel: [1914649.003141] Lustre: atlas2-MDT0000-lwp-OST00fe: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4e7.ccs.ornl.gov kernel: [1914649.005086] Lustre: 26092:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8802df138400 x1471904522621392/t0(0) o400->atlas2-MDT0000-lwp-OST033e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4e7.ccs.ornl.gov kernel: [1914649.005091] Lustre: 26092:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4e7.ccs.ornl.gov kernel: [1914649.005098] Lustre: atlas2-MDT0000-lwp-OST033e: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4e7.ccs.ornl.gov kernel: [1914649.005100] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3a1.ccs.ornl.gov kernel: [1914646.598632] Lustre: atlas2-MDT0000-lwp-OST0360: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3a1.ccs.ornl.gov kernel: [1914646.742616] Lustre: 22442:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3b5.ccs.ornl.gov kernel: [1914664.951370] Lustre: atlas2-MDT0000-lwp-OST02dc: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4b1.ccs.ornl.gov kernel: [1914656.439804] Lustre: 25872:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8802eef57000 x1471904507948948/t0(0) o400->atlas2-MDT0000-lwp-OST0050@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4i8.ccs.ornl.gov kernel: [1914665.493010] Lustre: 26096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880fc6b9f000 x1471904520246992/t0(0) o400->atlas2-MDT0000-lwp-OST01af@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4i8.ccs.ornl.gov kernel: [1914665.494170] Lustre: atlas2-MDT0000-lwp-OST011f: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4i8.ccs.ornl.gov kernel: [1914665.641690] Lustre: 26096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3f5.ccs.ornl.gov kernel: [801343.855415] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4a5.ccs.ornl.gov kernel: [1914670.537745] Lustre: 22123:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880b2de0b400 x1471904519454012/t0(0) o400->atlas2-MDT0000-lwp-OST016c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4a5.ccs.ornl.gov kernel: [1914670.537788] Lustre: atlas2-MDT0000-lwp-OST031c: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4a5.ccs.ornl.gov kernel: [1914670.537791] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4a5.ccs.ornl.gov kernel: [1914670.539738] Lustre: 22128:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880a07b90c00 x1471904519454028/t0(0) o400->atlas2-MDT0000-lwp-OST03ac@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4a5.ccs.ornl.gov kernel: [1914670.539743] Lustre: 22128:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4a5.ccs.ornl.gov kernel: [1914670.539752] Lustre: atlas2-MDT0000-lwp-OST03ac: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4a5.ccs.ornl.gov kernel: [1914670.539754] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4g7.ccs.ornl.gov kernel: [1914674.250412] Lustre: atlas2-MDT0000-lwp-OST022e: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3h2.ccs.ornl.gov kernel: [1914655.004318] Lustre: atlas2-MDT0000-lwp-OST0159: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4h8.ccs.ornl.gov kernel: [1914642.874568] Lustre: 24633:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3h4.ccs.ornl.gov kernel: [1914655.743107] Lustre: atlas2-MDT0000-lwp-OST015b: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g5.ccs.ornl.gov kernel: [1913253.551560] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4a2.ccs.ornl.gov kernel: [1914666.241245] Lustre: 25782:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880873527400 x1471904510895592/t0(0) o400->atlas2-MDT0000-lwp-OST03a9@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4a2.ccs.ornl.gov kernel: [1914666.241269] Lustre: atlas2-MDT0000-lwp-OST0169: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4a2.ccs.ornl.gov kernel: [1914666.268250] Lustre: 25792:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88055600ec00 x1471904510895568/t0(0) o400->atlas2-MDT0000-lwp-OST0049@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4a2.ccs.ornl.gov kernel: [1914666.268255] Lustre: 25792:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4a2.ccs.ornl.gov kernel: [1914666.268263] Lustre: atlas2-MDT0000-lwp-OST0049: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4a2.ccs.ornl.gov kernel: [1914666.268265] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4e7.ccs.ornl.gov kernel: [1914649.353311] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3a6.ccs.ornl.gov kernel: [1914654.602699] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4f5.ccs.ornl.gov kernel: [1914649.658828] Lustre: 22152:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff881015c6e400 x1471904509390796/t0(0) o400->atlas2-MDT0000-lwp-OST0224@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4f5.ccs.ornl.gov kernel: [1914649.659700] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880cda4c8800 x1471904509390792/t0(0) o400->atlas2-MDT0000-lwp-OST0104@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4f5.ccs.ornl.gov kernel: [1914649.659711] Lustre: atlas2-MDT0000-lwp-OST0104: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4f5.ccs.ornl.gov kernel: [1914649.897930] Lustre: 22152:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4i6.ccs.ornl.gov kernel: [1914665.209431] Lustre: 25730:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88086ba2a400 x1471904510597440/t0(0) o400->atlas2-MDT0000-lwp-OST011d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4i6.ccs.ornl.gov kernel: [1914665.209444] Lustre: atlas2-MDT0000-lwp-OST02cd: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4i6.ccs.ornl.gov kernel: [1914665.353268] Lustre: 25730:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3f8.ccs.ornl.gov kernel: [801354.364834] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4c2.ccs.ornl.gov kernel: [1914655.908864] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4g3.ccs.ornl.gov kernel: [1914662.965396] Lustre: 25832:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880a37bb7c00 x1471904517610096/t0(0) o400->atlas2-MDT0000-lwp-OST019a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4g3.ccs.ornl.gov kernel: [1914662.965403] Lustre: atlas2-MDT0000-lwp-OST034a: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4g3.ccs.ornl.gov kernel: [1914663.116423] Lustre: 25832:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss4g8.ccs.ornl.gov kernel: [1914650.733307] Lustre: 25553:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880543ea0c00 x1471904521746684/t0(0) o400->atlas2-MDT0000-lwp-OST007f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4g8.ccs.ornl.gov kernel: [1914650.733341] Lustre: atlas2-MDT0000-lwp-OST010f: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4g8.ccs.ornl.gov kernel: [1914650.737299] Lustre: 25557:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880250e22800 x1471904521746700/t0(0) o400->atlas2-MDT0000-lwp-OST02bf@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4g8.ccs.ornl.gov kernel: [1914650.737304] Lustre: 25557:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4g8.ccs.ornl.gov kernel: [1914650.737311] Lustre: atlas2-MDT0000-lwp-OST02bf: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4g8.ccs.ornl.gov kernel: [1914650.737313] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4g8.ccs.ornl.gov kernel: [1914651.082645] Lustre: 25553:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3a6.ccs.ornl.gov kernel: [1914654.622402] Lustre: 25782:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88031b3ef800 x1471904512394504/t0(0) o400->atlas2-MDT0000-lwp-OST0095@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4d4.ccs.ornl.gov kernel: [1914672.173626] Lustre: 27389:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88058196fc00 x1471904513679036/t0(0) o400->atlas2-MDT0000-lwp-OST03c3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4d4.ccs.ornl.gov kernel: [1914672.173640] Lustre: atlas2-MDT0000-lwp-OST0063: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g7.ccs.ornl.gov kernel: [1913249.343755] Lustre: 13462:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880870ba6c00 x1471904520422116/t0(0) o400->atlas2-MDT0000-lwp-OST0156@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3a1.ccs.ornl.gov kernel: [1914646.773075] Lustre: atlas2-MDT0000-lwp-OST0120: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4f5.ccs.ornl.gov kernel: [1914649.928179] Lustre: atlas2-MDT0000-lwp-OST0224: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b5.ccs.ornl.gov kernel: [1914665.010844] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4g3.ccs.ornl.gov kernel: [1914663.146773] Lustre: atlas2-MDT0000-lwp-OST019a: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3i4.ccs.ornl.gov kernel: [1914657.632722] Lustre: 27271:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880536893400 x1471904510470248/t0(0) o400->atlas2-MDT0000-lwp-OST01f3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3i4.ccs.ornl.gov kernel: [1914657.632727] Lustre: 27268:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88017dc20c00 x1471904510470236/t0(0) o400->atlas2-MDT0000-lwp-OST0043@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3i4.ccs.ornl.gov kernel: [1914657.632738] Lustre: atlas2-MDT0000-lwp-OST0043: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3i4.ccs.ornl.gov kernel: [1914657.876284] Lustre: 27271:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4b5.ccs.ornl.gov kernel: [1914669.420352] Lustre: 25795:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880585415c00 x1471904518365540/t0(0) o400->atlas2-MDT0000-lwp-OST03b4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4b5.ccs.ornl.gov kernel: [1914669.420364] Lustre: atlas2-MDT0000-lwp-OST0174: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4b5.ccs.ornl.gov kernel: [1914669.564349] Lustre: 25795:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4g7.ccs.ornl.gov kernel: [1914674.301624] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3h2.ccs.ornl.gov kernel: [1914655.054995] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3b5.ccs.ornl.gov kernel: [1914665.021964] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880416cf9000 x1471904514716840/t0(0) o400->atlas2-MDT0000-lwp-OST024c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4b6.ccs.ornl.gov kernel: [1914649.588215] Lustre: atlas2-MDT0000-lwp-OST00e5: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4b6.ccs.ornl.gov kernel: [1914649.639126] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4h8.ccs.ornl.gov kernel: [1914642.904884] Lustre: atlas2-MDT0000-lwp-OST03e7: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4h8.ccs.ornl.gov kernel: [1914642.964131] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3h3.ccs.ornl.gov kernel: [1914647.262035] Lustre: 26635:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880791304c00 x1471904512413468/t0(0) o400->atlas2-MDT0000-lwp-OST00ca@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3h3.ccs.ornl.gov kernel: [1914647.262052] Lustre: atlas2-MDT0000-lwp-OST01ea: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3h3.ccs.ornl.gov kernel: [1914647.264035] Lustre: 26625:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8804849ccc00 x1471904512413488/t0(0) o400->atlas2-MDT0000-lwp-OST039a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3h3.ccs.ornl.gov kernel: [1914647.264039] Lustre: 26625:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3h3.ccs.ornl.gov kernel: [1914647.264045] Lustre: atlas2-MDT0000-lwp-OST027a: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3h3.ccs.ornl.gov kernel: [1914647.264048] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3c7.ccs.ornl.gov kernel: [1914652.754990] Lustre: 25764:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880e1d59c800 x1471904510248556/t0(0) o400->atlas2-MDT0000-lwp-OST0376@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3c7.ccs.ornl.gov kernel: [1914652.755000] Lustre: atlas2-MDT0000-lwp-OST0136: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3c7.ccs.ornl.gov kernel: [1914652.901438] Lustre: 25764:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3h4.ccs.ornl.gov kernel: [1914655.802539] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4d4.ccs.ornl.gov kernel: [1914672.323550] Lustre: 27389:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3a1.ccs.ornl.gov kernel: [1914646.832670] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4g4.ccs.ornl.gov kernel: [1914666.295560] Lustre: 25817:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88074bf9b800 x1471904515530532/t0(0) o400->atlas2-MDT0000-lwp-OST019b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4g4.ccs.ornl.gov kernel: [1914666.295573] Lustre: atlas2-MDT0000-lwp-OST034b: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4g4.ccs.ornl.gov kernel: [1914666.295576] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4g4.ccs.ornl.gov kernel: [1914666.467385] Lustre: 25817:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4g4.ccs.ornl.gov kernel: [1914666.467420] Lustre: atlas2-MDT0000-lwp-OST010b: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4g4.ccs.ornl.gov kernel: [1914666.467423] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3f7.ccs.ornl.gov kernel: [800982.742744] Lustre: 78678:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880731bb8000 x1473076863306060/t0(0) o400->atlas2-MDT0000-lwp-OST01de@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3f7.ccs.ornl.gov kernel: [800982.742853] Lustre: atlas2-MDT0000-lwp-OST002e: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3f7.ccs.ornl.gov kernel: [800982.744739] Lustre: 78677:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8806cb217000 x1473076863306056/t0(0) o400->atlas2-MDT0000-lwp-OST014e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3f7.ccs.ornl.gov kernel: [800982.744743] Lustre: 78677:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3f7.ccs.ornl.gov kernel: [800982.744752] Lustre: atlas2-MDT0000-lwp-OST014e: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3f7.ccs.ornl.gov kernel: [800982.744754] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4e2.ccs.ornl.gov kernel: [1914665.588174] Lustre: 22173:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880a05878800 x1471904514146200/t0(0) o400->atlas2-MDT0000-lwp-OST0069@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4e2.ccs.ornl.gov kernel: [1914665.588178] Lustre: 22173:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4e2.ccs.ornl.gov kernel: [1914665.588187] Lustre: atlas2-MDT0000-lwp-OST0069: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4e2.ccs.ornl.gov kernel: [1914665.588189] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss4g7.ccs.ornl.gov kernel: [1914674.321123] Lustre: 26066:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88101244a400 x1471904508873444/t0(0) o400->atlas2-MDT0000-lwp-OST019e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3h2.ccs.ornl.gov kernel: [1914655.074883] Lustre: 20919:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880febd99c00 x1471904518760360/t0(0) o400->atlas2-MDT0000-lwp-OST00c9@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4b7.ccs.ornl.gov kernel: [1914645.823376] Lustre: 23168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880f67118c00 x1471904509377164/t0(0) o400->atlas2-MDT0000-lwp-OST0296@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4b7.ccs.ornl.gov kernel: [1914645.825389] Lustre: 23164:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880b11f9f800 x1471904509377148/t0(0) o400->atlas2-MDT0000-lwp-OST0056@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4b7.ccs.ornl.gov kernel: [1914645.825393] Lustre: 23164:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4b7.ccs.ornl.gov kernel: [1914645.825399] Lustre: atlas2-MDT0000-lwp-OST0176: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4b7.ccs.ornl.gov kernel: [1914646.097197] Lustre: 23168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4f3.ccs.ornl.gov kernel: [1914646.617267] Lustre: 26004:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880513adf000 x1471904513977020/t0(0) o400->atlas2-MDT0000-lwp-OST0222@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3h4.ccs.ornl.gov kernel: [1914655.813871] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88100b32fc00 x1471904509245124/t0(0) o400->atlas2-MDT0000-lwp-OST00cb@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4c1.ccs.ornl.gov kernel: [1914657.431438] Lustre: 22145:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88037637d000 x1471904516097376/t0(0) o400->atlas2-MDT0000-lwp-OST0208@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e5.ccs.ornl.gov kernel: [1914653.142902] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880425a4fc00 x1471904510654664/t0(0) o400->atlas2-MDT0000-lwp-OST00b4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e5.ccs.ornl.gov kernel: [1914653.142920] Lustre: atlas2-MDT0000-lwp-OST01d4: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e5.ccs.ornl.gov kernel: [1914653.286810] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4b5.ccs.ornl.gov kernel: [1914669.594919] Lustre: atlas2-MDT0000-lwp-OST03b4: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e4.ccs.ornl.gov kernel: [1914666.133353] Lustre: 22105:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8807c324b000 x1471904508537884/t0(0) o400->atlas2-MDT0000-lwp-OST00b3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e4.ccs.ornl.gov kernel: [1914666.133374] Lustre: atlas2-MDT0000-lwp-OST02f3: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e4.ccs.ornl.gov kernel: [1914666.282579] Lustre: 22107:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880264702c00 x1471904508537888/t0(0) o400->atlas2-MDT0000-lwp-OST0143@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e4.ccs.ornl.gov kernel: [1914666.282583] Lustre: 22105:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3e4.ccs.ornl.gov kernel: [1914666.282593] Lustre: atlas2-MDT0000-lwp-OST00b3: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e4.ccs.ornl.gov kernel: [1914666.282595] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3g7.ccs.ornl.gov kernel: [1913249.343793] Lustre: atlas2-MDT0000-lwp-OST01e6: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g7.ccs.ornl.gov kernel: [1913249.361727] Lustre: atlas2-MDT0000-lwp-OST0276: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g7.ccs.ornl.gov kernel: [1913249.361730] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3g7.ccs.ornl.gov kernel: [1913249.561953] Lustre: 13462:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4718294 previous similar messages Jul 17 16:44:57 atlas-oss3g2.ccs.ornl.gov kernel: [1913248.520747] Lustre: 14009:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8806ed8a7800 x1471904509035992/t0(0) o400->atlas2-MDT0000-lwp-OST01e1@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4d6.ccs.ornl.gov kernel: [1914651.982516] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8807622ce000 x1471904519610292/t0(0) o400->atlas2-MDT0000-lwp-OST03c5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3g2.ccs.ornl.gov kernel: [1913248.520757] Lustre: atlas2-MDT0000-lwp-OST0151: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g2.ccs.ornl.gov kernel: [1913248.520760] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3g2.ccs.ornl.gov kernel: [1913248.681533] Lustre: 14009:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4d6.ccs.ornl.gov kernel: [1914651.982542] Lustre: atlas2-MDT0000-lwp-OST00f5: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4d6.ccs.ornl.gov kernel: [1914652.134992] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3i4.ccs.ornl.gov kernel: [1914657.915058] Lustre: atlas2-MDT0000-lwp-OST01f3: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b3.ccs.ornl.gov kernel: [1914644.671841] Lustre: 25590:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8803b56ac000 x1471904513820296/t0(0) o400->atlas2-MDT0000-lwp-OST01ba@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3b3.ccs.ornl.gov kernel: [1914644.671895] Lustre: atlas2-MDT0000-lwp-OST009a: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b3.ccs.ornl.gov kernel: [1914644.674799] Lustre: 25592:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8803b56ac400 x1471904513820300/t0(0) o400->atlas2-MDT0000-lwp-OST024a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3b3.ccs.ornl.gov kernel: [1914644.674804] Lustre: 25592:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3b3.ccs.ornl.gov kernel: [1914644.674811] Lustre: atlas2-MDT0000-lwp-OST024a: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b3.ccs.ornl.gov kernel: [1914644.674813] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3g1.ccs.ornl.gov kernel: [1913241.873376] Lustre: 13834:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880cc163ec00 x1471904510299488/t0(0) o400->atlas2-MDT0000-lwp-OST0030@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3g1.ccs.ornl.gov kernel: [1913241.873389] Lustre: atlas2-MDT0000-lwp-OST0150: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g1.ccs.ornl.gov kernel: [1913242.017443] Lustre: 13834:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3a5.ccs.ornl.gov kernel: [1914645.247038] Lustre: 22236:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880748634800 x1471904511738332/t0(0) o400->atlas2-MDT0000-lwp-OST0094@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3a5.ccs.ornl.gov kernel: [1914645.247079] Lustre: atlas2-MDT0000-lwp-OST0004: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3a5.ccs.ornl.gov kernel: [1914645.248020] Lustre: 22234:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880357a9c400 x1471904511738328/t0(0) o400->atlas2-MDT0000-lwp-OST0124@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3a5.ccs.ornl.gov kernel: [1914645.248024] Lustre: 22234:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3a5.ccs.ornl.gov kernel: [1914645.248032] Lustre: atlas2-MDT0000-lwp-OST0124: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3a2.ccs.ornl.gov kernel: [1914651.444315] Lustre: 25623:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880f55072c00 x1471904510524156/t0(0) o400->atlas2-MDT0000-lwp-OST02d1@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3a2.ccs.ornl.gov kernel: [1914651.444332] Lustre: atlas2-MDT0000-lwp-OST0091: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3a2.ccs.ornl.gov kernel: [1914651.590999] Lustre: 25623:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss4b7.ccs.ornl.gov kernel: [1914646.127725] Lustre: atlas2-MDT0000-lwp-OST0296: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4f3.ccs.ornl.gov kernel: [1914646.617278] Lustre: atlas2-MDT0000-lwp-OST0102: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4f3.ccs.ornl.gov kernel: [1914646.765588] Lustre: 26004:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss4f5.ccs.ornl.gov kernel: [1914649.987823] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4g3.ccs.ornl.gov kernel: [1914663.206213] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3e5.ccs.ornl.gov kernel: [1914653.317200] Lustre: atlas2-MDT0000-lwp-OST00b4: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4d5.ccs.ornl.gov kernel: [1914664.427055] Lustre: 26096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880db6ede800 x1471904519136704/t0(0) o400->atlas2-MDT0000-lwp-OST03c4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4d5.ccs.ornl.gov kernel: [1914664.427066] Lustre: atlas2-MDT0000-lwp-OST00f4: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4d5.ccs.ornl.gov kernel: [1914664.574180] Lustre: 26096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss4e3.ccs.ornl.gov kernel: [1914657.719752] Lustre: 27362:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88050ba6f800 x1471904515523596/t0(0) o400->atlas2-MDT0000-lwp-OST033a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4e3.ccs.ornl.gov kernel: [1914657.719766] Lustre: atlas2-MDT0000-lwp-OST018a: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4e3.ccs.ornl.gov kernel: [1914657.719769] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4e3.ccs.ornl.gov kernel: [1914657.882772] Lustre: 27362:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss4d6.ccs.ornl.gov kernel: [1914652.165730] Lustre: atlas2-MDT0000-lwp-OST03c5: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4i5.ccs.ornl.gov kernel: [1914659.882286] Lustre: 25969:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8807846a0400 x1471904512720288/t0(0) o400->atlas2-MDT0000-lwp-OST03ec@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4i5.ccs.ornl.gov kernel: [1914659.882294] Lustre: atlas2-MDT0000-lwp-OST023c: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4i5.ccs.ornl.gov kernel: [1914660.029041] Lustre: 25969:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3g1.ccs.ornl.gov kernel: [1913242.047683] Lustre: atlas2-MDT0000-lwp-OST0030: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3a5.ccs.ornl.gov kernel: [1914645.583223] Lustre: 22236:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3c8.ccs.ornl.gov kernel: [1914656.979992] Lustre: 26028:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880b8a9a2000 x1471904507927904/t0(0) o400->atlas2-MDT0000-lwp-OST00a7@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3c8.ccs.ornl.gov kernel: [1914656.980018] Lustre: atlas2-MDT0000-lwp-OST0377: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3c8.ccs.ornl.gov kernel: [1914657.131635] Lustre: 26028:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3g2.ccs.ornl.gov kernel: [1913248.720504] Lustre: atlas2-MDT0000-lwp-OST01e1: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3i8.ccs.ornl.gov kernel: [1914665.831486] Lustre: 22183:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88021b90a800 x1471904515355344/t0(0) o400->atlas2-MDT0000-lwp-OST0287@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3i8.ccs.ornl.gov kernel: [1914665.831514] Lustre: atlas2-MDT0000-lwp-OST0167: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3i8.ccs.ornl.gov kernel: [1914665.975511] Lustre: 22183:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3a2.ccs.ornl.gov kernel: [1914651.621526] Lustre: atlas2-MDT0000-lwp-OST02d1: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4h5.ccs.ornl.gov kernel: [1914654.443301] Lustre: 25683:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880073f35c00 x1471904509313324/t0(0) o400->atlas2-MDT0000-lwp-OST0114@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4h5.ccs.ornl.gov kernel: [1914654.443309] Lustre: atlas2-MDT0000-lwp-OST0234: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3h6.ccs.ornl.gov kernel: [1914652.324190] Lustre: 25747:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880be0e6b800 x1471904512216388/t0(0) o400->atlas2-MDT0000-lwp-OST015d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3h6.ccs.ornl.gov kernel: [1914652.324202] Lustre: atlas2-MDT0000-lwp-OST030d: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3h6.ccs.ornl.gov kernel: [1914652.467778] Lustre: 25747:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3d8.ccs.ornl.gov kernel: [1914658.480605] Lustre: 24582:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880bd8074000 x1471904508710544/t0(0) o400->atlas2-MDT0000-lwp-OST025f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3d8.ccs.ornl.gov kernel: [1914658.480625] Lustre: atlas2-MDT0000-lwp-OST02ef: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3d8.ccs.ornl.gov kernel: [1914658.480628] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3d8.ccs.ornl.gov kernel: [1914658.648838] Lustre: 24582:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3a4.ccs.ornl.gov kernel: [1914641.944111] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8806eb1b9400 x1471904512538664/t0(0) o400->atlas2-MDT0000-lwp-OST0363@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3a4.ccs.ornl.gov kernel: [1914641.944126] Lustre: atlas2-MDT0000-lwp-OST0243: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3a4.ccs.ornl.gov kernel: [1914642.087986] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3i4.ccs.ornl.gov kernel: [1914657.965987] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4d5.ccs.ornl.gov kernel: [1914664.604699] Lustre: atlas2-MDT0000-lwp-OST03c4: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4b5.ccs.ornl.gov kernel: [1914669.654400] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3f2.ccs.ornl.gov kernel: [1914647.092371] Lustre: 25723:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880b7f219400 x1471904513672388/t0(0) o400->atlas2-MDT0000-lwp-OST0149@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3f2.ccs.ornl.gov kernel: [1914647.092377] Lustre: atlas2-MDT0000-lwp-OST01d9: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3f2.ccs.ornl.gov kernel: [1914647.240958] Lustre: 25723:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 743173 previous similar messages Jul 17 16:44:57 atlas-oss3g6.ccs.ornl.gov kernel: [1913260.325758] Lustre: 13988:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8803d3a1c800 x1471904523473684/t0(0) o400->atlas2-MDT0000-lwp-OST0035@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3g6.ccs.ornl.gov kernel: [1913260.325769] Lustre: atlas2-MDT0000-lwp-OST00c5: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g6.ccs.ornl.gov kernel: [1913260.470321] Lustre: 13988:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3g6.ccs.ornl.gov kernel: [1913260.470364] Lustre: 13990:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88014069d000 x1471904523473692/t0(0) o400->atlas2-MDT0000-lwp-OST0155@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3g6.ccs.ornl.gov kernel: [1913260.470368] Lustre: 13990:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3g6.ccs.ornl.gov kernel: [1913260.470378] Lustre: atlas2-MDT0000-lwp-OST0155: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g6.ccs.ornl.gov kernel: [1913260.470380] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3d2.ccs.ornl.gov kernel: [1914653.663625] Lustre: 25988:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880626a31000 x1471904511214520/t0(0) o400->atlas2-MDT0000-lwp-OST0259@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3d2.ccs.ornl.gov kernel: [1914653.663653] Lustre: atlas2-MDT0000-lwp-OST0379: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3d2.ccs.ornl.gov kernel: [1914653.663656] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3d2.ccs.ornl.gov kernel: [1914653.824227] Lustre: 25986:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8804eb30dc00 x1471904511214512/t0(0) o400->atlas2-MDT0000-lwp-OST0139@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4e3.ccs.ornl.gov kernel: [1914657.921237] Lustre: atlas2-MDT0000-lwp-OST033a: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3d1.ccs.ornl.gov kernel: [1914644.628732] Lustre: 25667:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880184ef0800 x1471904515778192/t0(0) o400->atlas2-MDT0000-lwp-OST02e8@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3d1.ccs.ornl.gov kernel: [1914644.628747] Lustre: atlas2-MDT0000-lwp-OST0258: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3d1.ccs.ornl.gov kernel: [1914644.628751] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3d1.ccs.ornl.gov kernel: [1914644.783823] Lustre: 25667:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss4i5.ccs.ornl.gov kernel: [1914660.059641] Lustre: atlas2-MDT0000-lwp-OST03ec: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4h1.ccs.ornl.gov kernel: [1914645.848469] Lustre: 25865:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880e39bc3000 x1471904511602988/t0(0) o400->atlas2-MDT0000-lwp-OST01a0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4h1.ccs.ornl.gov kernel: [1914645.848485] Lustre: atlas2-MDT0000-lwp-OST02c0: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4h1.ccs.ornl.gov kernel: [1914645.998039] Lustre: 25865:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3c4.ccs.ornl.gov kernel: [1914647.260658] Lustre: 25962:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8805312ddc00 x1471904511832796/t0(0) o400->atlas2-MDT0000-lwp-OST02e3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3c8.ccs.ornl.gov kernel: [1914657.162085] Lustre: atlas2-MDT0000-lwp-OST00a7: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b7.ccs.ornl.gov kernel: [1914656.854155] Lustre: 26035:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88045e432000 x1471904519392520/t0(0) o400->atlas2-MDT0000-lwp-OST036e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3i8.ccs.ornl.gov kernel: [1914666.005989] Lustre: atlas2-MDT0000-lwp-OST0287: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4e5.ccs.ornl.gov kernel: [1914653.057910] Lustre: 24651:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88017d0cfc00 x1471904515279084/t0(0) o400->atlas2-MDT0000-lwp-OST033c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4e5.ccs.ornl.gov kernel: [1914653.057920] Lustre: atlas2-MDT0000-lwp-OST03cc: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4b5.ccs.ornl.gov kernel: [1914669.673611] Lustre: 25795:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8803fea78c00 x1471904518365536/t0(0) o400->atlas2-MDT0000-lwp-OST0294@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e5.ccs.ornl.gov kernel: [1914653.368424] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4f6.ccs.ornl.gov kernel: [1914646.654468] Lustre: 22291:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880d21e8bc00 x1471904511708436/t0(0) o400->atlas2-MDT0000-lwp-OST03d5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4f6.ccs.ornl.gov kernel: [1914646.654472] Lustre: 22291:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4f6.ccs.ornl.gov kernel: [1914646.654482] Lustre: atlas2-MDT0000-lwp-OST03d5: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4f6.ccs.ornl.gov kernel: [1914646.654484] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3a3.ccs.ornl.gov kernel: [1914649.200495] Lustre: 27253:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff881008a59800 x1471904526379780/t0(0) o400->atlas2-MDT0000-lwp-OST0122@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3a3.ccs.ornl.gov kernel: [1914649.200502] Lustre: atlas2-MDT0000-lwp-OST0242: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3a3.ccs.ornl.gov kernel: [1914649.200506] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3a3.ccs.ornl.gov kernel: [1914649.363098] Lustre: 27253:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3b6.ccs.ornl.gov kernel: [1914650.211509] Lustre: 25962:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880120636400 x1471904512275240/t0(0) o400->atlas2-MDT0000-lwp-OST009d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3b6.ccs.ornl.gov kernel: [1914650.211540] Lustre: atlas2-MDT0000-lwp-OST012d: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b6.ccs.ornl.gov kernel: [1914650.357240] Lustre: 25962:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3e8.ccs.ornl.gov kernel: [1914652.077917] Lustre: 25793:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8805580d1400 x1471904515139452/t0(0) o400->atlas2-MDT0000-lwp-OST02f7@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e8.ccs.ornl.gov kernel: [1914652.077927] Lustre: atlas2-MDT0000-lwp-OST01d7: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e8.ccs.ornl.gov kernel: [1914652.077933] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3e8.ccs.ornl.gov kernel: [1914652.247747] Lustre: 25793:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss4b7.ccs.ornl.gov kernel: [1914646.187179] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4f3.ccs.ornl.gov kernel: [1914646.796191] Lustre: atlas2-MDT0000-lwp-OST0222: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3d8.ccs.ornl.gov kernel: [1914658.679196] Lustre: atlas2-MDT0000-lwp-OST025f: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4h5.ccs.ornl.gov kernel: [1914654.589662] Lustre: 25683:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3h6.ccs.ornl.gov kernel: [1914652.506907] Lustre: atlas2-MDT0000-lwp-OST015d: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4h5.ccs.ornl.gov kernel: [1914654.629831] Lustre: atlas2-MDT0000-lwp-OST0114: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b2.ccs.ornl.gov kernel: [1914651.986399] Lustre: 25285:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880a04b1b000 x1471904511916860/t0(0) o400->atlas2-MDT0000-lwp-OST0009@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3b2.ccs.ornl.gov kernel: [1914651.986557] Lustre: atlas2-MDT0000-lwp-OST0249: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b2.ccs.ornl.gov kernel: [1914652.129617] Lustre: 25285:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3i1.ccs.ornl.gov kernel: [1914652.135727] Lustre: 22138:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88045ba76400 x1471904519755272/t0(0) o400->atlas2-MDT0000-lwp-OST01f0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3i1.ccs.ornl.gov kernel: [1914652.135747] Lustre: atlas2-MDT0000-lwp-OST0040: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3i1.ccs.ornl.gov kernel: [1914652.279763] Lustre: 22138:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4g6.ccs.ornl.gov kernel: [1914661.423492] Lustre: atlas2-MDT0000-lwp-OST007d: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4g6.ccs.ornl.gov kernel: [1914661.571366] Lustre: 24695:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss4g6.ccs.ornl.gov kernel: [1914661.571401] Lustre: 24691:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff88057c380800 x1471904507898552/t0(0) o400->atlas2-MDT0000-lwp-OST022d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4g6.ccs.ornl.gov kernel: [1914661.571405] Lustre: 24691:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss4g6.ccs.ornl.gov kernel: [1914661.571414] Lustre: atlas2-MDT0000-lwp-OST022d: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4g6.ccs.ornl.gov kernel: [1914661.571417] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4i1.ccs.ornl.gov kernel: [1914652.093508] Lustre: 25775:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880b8d601000 x1471904513319588/t0(0) o400->atlas2-MDT0000-lwp-OST0118@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4i1.ccs.ornl.gov kernel: [1914652.093525] Lustre: atlas2-MDT0000-lwp-OST0238: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4i1.ccs.ornl.gov kernel: [1914652.093529] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4i1.ccs.ornl.gov kernel: [1914652.254527] Lustre: 25775:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3a4.ccs.ornl.gov kernel: [1914642.126833] Lustre: atlas2-MDT0000-lwp-OST0363: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g2.ccs.ornl.gov kernel: [1913248.771599] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4f3.ccs.ornl.gov kernel: [1914646.855476] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4d6.ccs.ornl.gov kernel: [1914652.225198] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3g4.ccs.ornl.gov kernel: [1913262.476358] Lustre: 13609:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff881068537400 x1471904510791220/t0(0) o400->atlas2-MDT0000-lwp-OST00c3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3g4.ccs.ornl.gov kernel: [1913262.476374] Lustre: atlas2-MDT0000-lwp-OST0393: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g4.ccs.ornl.gov kernel: [1913262.628863] Lustre: 13609:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3h1.ccs.ornl.gov kernel: [1914640.502967] Lustre: 22216:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8803f66aac00 x1471904517765760/t0(0) o400->atlas2-MDT0000-lwp-OST00c8@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3c2.ccs.ornl.gov kernel: [1914654.047948] Lustre: 25863:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8804a8b0c000 x1471904511647808/t0(0) o400->atlas2-MDT0000-lwp-OST0251@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3h1.ccs.ornl.gov kernel: [1914640.502979] Lustre: atlas2-MDT0000-lwp-OST0158: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3h1.ccs.ornl.gov kernel: [1914640.505915] Lustre: 22206:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8805f46bb400 x1471904517765780/t0(0) o400->atlas2-MDT0000-lwp-OST0398@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3h1.ccs.ornl.gov kernel: [1914640.505920] Lustre: 22206:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3h1.ccs.ornl.gov kernel: [1914640.505928] Lustre: atlas2-MDT0000-lwp-OST0398: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3h1.ccs.ornl.gov kernel: [1914640.505930] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3h1.ccs.ornl.gov kernel: [1914640.854514] Lustre: 22216:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3g1.ccs.ornl.gov kernel: [1913242.107059] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3c2.ccs.ornl.gov kernel: [1914654.047961] Lustre: atlas2-MDT0000-lwp-OST01c1: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3c2.ccs.ornl.gov kernel: [1914654.192147] Lustre: 25863:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3e5.ccs.ornl.gov kernel: [1914653.388096] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88066af71400 x1471904510654660/t0(0) o400->atlas2-MDT0000-lwp-OST0024@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4e8.ccs.ornl.gov kernel: [1914664.742564] Lustre: 24607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880825086c00 x1471904518472660/t0(0) o400->atlas2-MDT0000-lwp-OST018f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3f3.ccs.ornl.gov kernel: [1914665.499192] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880aec69a800 x1471904510687964/t0(0) o400->atlas2-MDT0000-lwp-OST026a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3c4.ccs.ornl.gov kernel: [1914647.260667] Lustre: atlas2-MDT0000-lwp-OST0373: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3c4.ccs.ornl.gov kernel: [1914647.406169] Lustre: 25962:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4e8.ccs.ornl.gov kernel: [1914664.742611] Lustre: atlas2-MDT0000-lwp-OST006f: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4e8.ccs.ornl.gov kernel: [1914664.744539] Lustre: 24595:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8801e2273400 x1471904518472676/t0(0) o400->atlas2-MDT0000-lwp-OST03cf@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4e8.ccs.ornl.gov kernel: [1914664.744543] Lustre: 24595:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss4e8.ccs.ornl.gov kernel: [1914664.744551] Lustre: atlas2-MDT0000-lwp-OST02af: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4e8.ccs.ornl.gov kernel: [1914664.744554] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss4e8.ccs.ornl.gov kernel: [1914665.095860] Lustre: 24607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4b8.ccs.ornl.gov kernel: [1914653.645764] Lustre: 25795:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880bf39d3400 x1471904512563792/t0(0) o400->atlas2-MDT0000-lwp-OST03b7@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4b8.ccs.ornl.gov kernel: [1914653.645828] Lustre: atlas2-MDT0000-lwp-OST0177: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4b8.ccs.ornl.gov kernel: [1914653.648708] Lustre: 25804:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880de3ff0400 x1471904512563768/t0(0) o400->atlas2-MDT0000-lwp-OST0057@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4b8.ccs.ornl.gov kernel: [1914653.648712] Lustre: 25804:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4b8.ccs.ornl.gov kernel: [1914653.648719] Lustre: atlas2-MDT0000-lwp-OST0057: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4b8.ccs.ornl.gov kernel: [1914653.648721] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3c5.ccs.ornl.gov kernel: [1914658.609206] Lustre: 25804:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880bca00b800 x1471904512465996/t0(0) o400->atlas2-MDT0000-lwp-OST0254@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3a2.ccs.ornl.gov kernel: [1914651.680768] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3d1.ccs.ornl.gov kernel: [1914644.822699] Lustre: atlas2-MDT0000-lwp-OST02e8: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4e5.ccs.ornl.gov kernel: [1914653.204228] Lustre: 24651:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11736293 previous similar messages Jul 17 16:44:57 atlas-oss4e5.ccs.ornl.gov kernel: [1914653.234866] Lustre: atlas2-MDT0000-lwp-OST033c: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4h1.ccs.ornl.gov kernel: [1914646.037071] Lustre: atlas2-MDT0000-lwp-OST01a0: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b6.ccs.ornl.gov kernel: [1914650.387843] Lustre: atlas2-MDT0000-lwp-OST009d: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e8.ccs.ornl.gov kernel: [1914652.278147] Lustre: atlas2-MDT0000-lwp-OST02f7: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4d6.ccs.ornl.gov kernel: [1914652.244840] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880612755400 x1471904519610288/t0(0) o400->atlas2-MDT0000-lwp-OST02a5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4d5.ccs.ornl.gov kernel: [1914664.664045] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4e3.ccs.ornl.gov kernel: [1914657.972510] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3b2.ccs.ornl.gov kernel: [1914652.160070] Lustre: atlas2-MDT0000-lwp-OST0009: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3a3.ccs.ornl.gov kernel: [1914649.402039] Lustre: atlas2-MDT0000-lwp-OST0122: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3c6.ccs.ornl.gov kernel: [1914644.279458] Lustre: 24544:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880940e12000 x1471904508186748/t0(0) o400->atlas2-MDT0000-lwp-OST0255@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3c8.ccs.ornl.gov kernel: [1914657.213287] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3c6.ccs.ornl.gov kernel: [1914644.279490] Lustre: atlas2-MDT0000-lwp-OST0375: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3c6.ccs.ornl.gov kernel: [1914644.424785] Lustre: 24544:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3c2.ccs.ornl.gov kernel: [1914654.222347] Lustre: atlas2-MDT0000-lwp-OST0251: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4i5.ccs.ornl.gov kernel: [1914660.119487] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4c7.ccs.ornl.gov kernel: [1914651.803580] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8802e58d5400 x1471904514425480/t0(0) o400->atlas2-MDT0000-lwp-OST017e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4c7.ccs.ornl.gov kernel: [1914651.803612] Lustre: atlas2-MDT0000-lwp-OST03be: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4c7.ccs.ornl.gov kernel: [1914651.803616] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4c7.ccs.ornl.gov kernel: [1914651.805575] Lustre: atlas2-MDT0000-lwp-OST00ee: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4c7.ccs.ornl.gov kernel: [1914651.805578] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4c7.ccs.ornl.gov kernel: [1914652.042715] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 18134260 previous similar messages Jul 17 16:44:57 atlas-oss4i1.ccs.ornl.gov kernel: [1914652.293623] Lustre: atlas2-MDT0000-lwp-OST0118: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4c6.ccs.ornl.gov kernel: [1914652.797947] Lustre: 22198:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff881072961400 x1471904517937096/t0(0) o400->atlas2-MDT0000-lwp-OST029d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4c6.ccs.ornl.gov kernel: [1914652.797952] Lustre: 22194:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880dfea70400 x1471904517937080/t0(0) o400->atlas2-MDT0000-lwp-OST005d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4c6.ccs.ornl.gov kernel: [1914652.797960] Lustre: 22194:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4c6.ccs.ornl.gov kernel: [1914652.797967] Lustre: atlas2-MDT0000-lwp-OST020d: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4c6.ccs.ornl.gov kernel: [1914652.797970] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4c6.ccs.ornl.gov kernel: [1914653.086473] Lustre: 22198:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3f3.ccs.ornl.gov kernel: [1914665.499203] Lustre: atlas2-MDT0000-lwp-OST038a: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3f3.ccs.ornl.gov kernel: [1914665.499206] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3f3.ccs.ornl.gov kernel: [1914665.670094] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4145518 previous similar messages Jul 17 16:44:57 atlas-oss3g4.ccs.ornl.gov kernel: [1913262.668690] Lustre: atlas2-MDT0000-lwp-OST00c3: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3i8.ccs.ornl.gov kernel: [1914666.065425] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3c5.ccs.ornl.gov kernel: [1914658.609212] Lustre: atlas2-MDT0000-lwp-OST0134: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3c5.ccs.ornl.gov kernel: [1914658.756292] Lustre: 25804:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss4h5.ccs.ornl.gov kernel: [1914654.680882] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4f1.ccs.ornl.gov kernel: [1914656.006813] Lustre: 25961:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8806ddf4e400 x1471904513370932/t0(0) o400->atlas2-MDT0000-lwp-OST0220@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4i8.ccs.ornl.gov kernel: [1914665.642095] Lustre: 26097:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880b503fb400 x1471904520246996/t0(0) o400->atlas2-MDT0000-lwp-OST023f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4f1.ccs.ornl.gov kernel: [1914656.006839] Lustre: atlas2-MDT0000-lwp-OST0340: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4f1.ccs.ornl.gov kernel: [1914656.008806] Lustre: 25959:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8802d9974c00 x1471904513370920/t0(0) o400->atlas2-MDT0000-lwp-OST0070@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4f1.ccs.ornl.gov kernel: [1914656.008810] Lustre: 25959:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4f1.ccs.ornl.gov kernel: [1914656.008819] Lustre: atlas2-MDT0000-lwp-OST0070: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4f1.ccs.ornl.gov kernel: [1914656.008823] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss4i8.ccs.ornl.gov kernel: [1914665.642099] Lustre: 26097:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4i8.ccs.ornl.gov kernel: [1914665.642107] Lustre: atlas2-MDT0000-lwp-OST023f: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4i8.ccs.ornl.gov kernel: [1914665.642109] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3c8.ccs.ornl.gov kernel: [1914657.232972] Lustre: 26028:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880eacdc4c00 x1471904507927900/t0(0) o400->atlas2-MDT0000-lwp-OST0017@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3d8.ccs.ornl.gov kernel: [1914658.738659] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3a4.ccs.ornl.gov kernel: [1914642.177658] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4f4.ccs.ornl.gov kernel: [1914652.300975] Lustre: 22157:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8803b40d0400 x1471904516980120/t0(0) o400->atlas2-MDT0000-lwp-OST0103@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3f2.ccs.ornl.gov kernel: [1914647.271758] Lustre: atlas2-MDT0000-lwp-OST0149: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3f2.ccs.ornl.gov kernel: [1914647.330984] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3c6.ccs.ornl.gov kernel: [1914644.455440] Lustre: atlas2-MDT0000-lwp-OST0255: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3d1.ccs.ornl.gov kernel: [1914644.873604] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3h6.ccs.ornl.gov kernel: [1914652.558080] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3h6.ccs.ornl.gov kernel: [1914652.577831] Lustre: 25747:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880be0e6b400 x1471904512216392/t0(0) o400->atlas2-MDT0000-lwp-OST01ed@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4h1.ccs.ornl.gov kernel: [1914646.088024] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3c4.ccs.ornl.gov kernel: [1914647.436794] Lustre: atlas2-MDT0000-lwp-OST02e3: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b6.ccs.ornl.gov kernel: [1914650.438590] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4a6.ccs.ornl.gov kernel: [1914653.928087] Lustre: 25594:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88013c863000 x1471904512502644/t0(0) o400->atlas2-MDT0000-lwp-OST028d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4a6.ccs.ornl.gov kernel: [1914653.928216] Lustre: atlas2-MDT0000-lwp-OST004d: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4a6.ccs.ornl.gov kernel: [1914653.930079] Lustre: 25591:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8803b920f000 x1471904512502636/t0(0) o400->atlas2-MDT0000-lwp-OST016d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4a6.ccs.ornl.gov kernel: [1914653.930084] Lustre: 25591:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss4a6.ccs.ornl.gov kernel: [1914653.930092] Lustre: atlas2-MDT0000-lwp-OST031d: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4a6.ccs.ornl.gov kernel: [1914653.930095] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3a3.ccs.ornl.gov kernel: [1914649.452974] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4c6.ccs.ornl.gov kernel: [1914653.125419] Lustre: atlas2-MDT0000-lwp-OST029d: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e8.ccs.ornl.gov kernel: [1914652.337403] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3c4.ccs.ornl.gov kernel: [1914647.496014] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4i6.ccs.ornl.gov kernel: [1914665.353349] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880555762400 x1471904510597460/t0(0) o400->atlas2-MDT0000-lwp-OST035d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4i6.ccs.ornl.gov kernel: [1914665.353353] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4i6.ccs.ornl.gov kernel: [1914665.353361] Lustre: atlas2-MDT0000-lwp-OST035d: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4i6.ccs.ornl.gov kernel: [1914665.353364] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3b2.ccs.ornl.gov kernel: [1914652.219457] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4i1.ccs.ornl.gov kernel: [1914652.344347] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4f4.ccs.ornl.gov kernel: [1914652.300987] Lustre: atlas2-MDT0000-lwp-OST0223: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4f4.ccs.ornl.gov kernel: [1914652.302967] Lustre: 22162:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff8805bf67bc00 x1471904516980136/t0(0) o400->atlas2-MDT0000-lwp-OST03d3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4f4.ccs.ornl.gov kernel: [1914652.302972] Lustre: 22162:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4f4.ccs.ornl.gov kernel: [1914652.302979] Lustre: atlas2-MDT0000-lwp-OST03d3: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4f4.ccs.ornl.gov kernel: [1914652.302981] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss4f4.ccs.ornl.gov kernel: [1914652.646124] Lustre: 22157:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3g4.ccs.ornl.gov kernel: [1913262.719597] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4e5.ccs.ornl.gov kernel: [1914653.294233] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3c2.ccs.ornl.gov kernel: [1914654.282133] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3f3.ccs.ornl.gov kernel: [1914665.700903] Lustre: atlas2-MDT0000-lwp-OST026a: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e6.ccs.ornl.gov kernel: [1914649.319883] Lustre: 22108:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880b4be2b400 x1471904520534504/t0(0) o400->atlas2-MDT0000-lwp-OST01d5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e6.ccs.ornl.gov kernel: [1914649.319902] Lustre: atlas2-MDT0000-lwp-OST02f5: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e6.ccs.ornl.gov kernel: [1914649.321797] Lustre: 22105:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880a08a88000 x1471904520534496/t0(0) o400->atlas2-MDT0000-lwp-OST00b5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e6.ccs.ornl.gov kernel: [1914649.321802] Lustre: 22107:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629329/real 1405629329] req@ffff880f98b5c400 x1471904520534500/t0(0) o400->atlas2-MDT0000-lwp-OST0145@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629896 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e6.ccs.ornl.gov kernel: [1914649.321806] Lustre: 22105:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3e6.ccs.ornl.gov kernel: [1914649.321808] Lustre: 22107:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3e6.ccs.ornl.gov kernel: [1914649.321815] Lustre: atlas2-MDT0000-lwp-OST00b5: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e6.ccs.ornl.gov kernel: [1914649.321817] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3e6.ccs.ornl.gov kernel: [1914649.808023] Lustre: 22108:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4a8.ccs.ornl.gov kernel: [1914658.003214] Lustre: 25967:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880cc59f9000 x1471904507750624/t0(0) o400->atlas2-MDT0000-lwp-OST00df@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4a8.ccs.ornl.gov kernel: [1914658.004188] Lustre: 25954:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880e546a6c00 x1471904507750640/t0(0) o400->atlas2-MDT0000-lwp-OST031f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4a8.ccs.ornl.gov kernel: [1914658.004193] Lustre: 25954:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4a8.ccs.ornl.gov kernel: [1914658.004197] Lustre: atlas2-MDT0000-lwp-OST004f: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4a8.ccs.ornl.gov kernel: [1914658.281034] Lustre: 25967:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3c5.ccs.ornl.gov kernel: [1914658.786932] Lustre: atlas2-MDT0000-lwp-OST0254: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4a6.ccs.ornl.gov kernel: [1914654.274282] Lustre: 25594:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss4h3.ccs.ornl.gov kernel: [1914664.643935] Lustre: 26063:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88022ff28c00 x1471904509650720/t0(0) o400->atlas2-MDT0000-lwp-OST0352@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4h3.ccs.ornl.gov kernel: [1914664.643940] Lustre: 26062:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88022ff28000 x1471904509650716/t0(0) o400->atlas2-MDT0000-lwp-OST0232@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4h3.ccs.ornl.gov kernel: [1914664.643944] Lustre: 26062:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4h3.ccs.ornl.gov kernel: [1914664.643947] Lustre: atlas2-MDT0000-lwp-OST03e2: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4h3.ccs.ornl.gov kernel: [1914664.923275] Lustre: 26063:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4i1.ccs.ornl.gov kernel: [1914652.364203] Lustre: 25775:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880bd8e47400 x1471904513319592/t0(0) o400->atlas2-MDT0000-lwp-OST01a8@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e4.ccs.ornl.gov kernel: [1914666.483248] Lustre: 22107:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3d3.ccs.ornl.gov kernel: [1914658.702263] Lustre: 25556:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8804dece2400 x1471904510125580/t0(0) o400->atlas2-MDT0000-lwp-OST00aa@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3d3.ccs.ornl.gov kernel: [1914658.702351] Lustre: atlas2-MDT0000-lwp-OST025a: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3d3.ccs.ornl.gov kernel: [1914658.702354] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3d3.ccs.ornl.gov kernel: [1914658.709235] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880236046800 x1471904510125596/t0(0) o400->atlas2-MDT0000-lwp-OST02ea@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3d3.ccs.ornl.gov kernel: [1914658.709240] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3d3.ccs.ornl.gov kernel: [1914658.709248] Lustre: atlas2-MDT0000-lwp-OST01ca: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3d3.ccs.ornl.gov kernel: [1914658.709250] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3d3.ccs.ornl.gov kernel: [1914659.071018] Lustre: 25556:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss4g1.ccs.ornl.gov kernel: [1914648.984024] Lustre: 25815:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88035c5aa400 x1471904508970268/t0(0) o400->atlas2-MDT0000-lwp-OST03d8@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3c5.ccs.ornl.gov kernel: [1914658.846311] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4g1.ccs.ornl.gov kernel: [1914648.984034] Lustre: atlas2-MDT0000-lwp-OST0078: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4g1.ccs.ornl.gov kernel: [1914648.989008] Lustre: 25812:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8807052a8800 x1471904508970256/t0(0) o400->atlas2-MDT0000-lwp-OST0228@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4g1.ccs.ornl.gov kernel: [1914648.989012] Lustre: 25812:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4g1.ccs.ornl.gov kernel: [1914648.989019] Lustre: atlas2-MDT0000-lwp-OST0228: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4g1.ccs.ornl.gov kernel: [1914648.989021] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4a1.ccs.ornl.gov kernel: [1914649.858217] Lustre: 25993:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880ecb678400 x1471904510345676/t0(0) o400->atlas2-MDT0000-lwp-OST0288@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4a1.ccs.ornl.gov kernel: [1914649.858343] Lustre: atlas2-MDT0000-lwp-OST0168: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4a1.ccs.ornl.gov kernel: [1914649.860226] Lustre: 25994:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880feee51c00 x1471904510345684/t0(0) o400->atlas2-MDT0000-lwp-OST03a8@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4a1.ccs.ornl.gov kernel: [1914649.860230] Lustre: 25994:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4a1.ccs.ornl.gov kernel: [1914649.860240] Lustre: atlas2-MDT0000-lwp-OST03a8: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4a1.ccs.ornl.gov kernel: [1914649.860242] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3f3.ccs.ornl.gov kernel: [1914665.760345] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3c6.ccs.ornl.gov kernel: [1914644.514833] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4a7.ccs.ornl.gov kernel: [1914663.559848] Lustre: 25805:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880536440c00 x1471904510935876/t0(0) o400->atlas2-MDT0000-lwp-OST004e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4a7.ccs.ornl.gov kernel: [1914663.559876] Lustre: atlas2-MDT0000-lwp-OST03ae: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4a7.ccs.ornl.gov kernel: [1914663.561802] Lustre: 25811:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88066a0c6c00 x1471904510935896/t0(0) o400->atlas2-MDT0000-lwp-OST031e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4a7.ccs.ornl.gov kernel: [1914663.561806] Lustre: 25811:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4a7.ccs.ornl.gov kernel: [1914663.561815] Lustre: atlas2-MDT0000-lwp-OST031e: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4a7.ccs.ornl.gov kernel: [1914663.561817] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4a7.ccs.ornl.gov kernel: [1914663.912028] Lustre: 25805:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4c6.ccs.ornl.gov kernel: [1914653.176494] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4e6.ccs.ornl.gov kernel: [1914667.664745] Lustre: 25704:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8801eef12c00 x1471904508407988/t0(0) o400->atlas2-MDT0000-lwp-OST021d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4e6.ccs.ornl.gov kernel: [1914667.664758] Lustre: atlas2-MDT0000-lwp-OST03cd: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4e6.ccs.ornl.gov kernel: [1914667.667744] Lustre: 25705:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88028ce1e400 x1471904508407992/t0(0) o400->atlas2-MDT0000-lwp-OST033d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4e6.ccs.ornl.gov kernel: [1914667.667748] Lustre: 25705:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4e6.ccs.ornl.gov kernel: [1914667.667755] Lustre: atlas2-MDT0000-lwp-OST033d: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4e6.ccs.ornl.gov kernel: [1914667.667757] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss4e6.ccs.ornl.gov kernel: [1914668.011044] Lustre: 25704:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3c7.ccs.ornl.gov kernel: [1914652.901515] Lustre: 25762:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880a599f1000 x1471904510248548/t0(0) o400->atlas2-MDT0000-lwp-OST0256@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3c7.ccs.ornl.gov kernel: [1914652.901519] Lustre: 25762:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3c7.ccs.ornl.gov kernel: [1914652.901527] Lustre: atlas2-MDT0000-lwp-OST0256: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3c7.ccs.ornl.gov kernel: [1914652.901529] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss4a8.ccs.ornl.gov kernel: [1914658.311626] Lustre: atlas2-MDT0000-lwp-OST00df: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4h3.ccs.ornl.gov kernel: [1914664.953894] Lustre: atlas2-MDT0000-lwp-OST0352: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e6.ccs.ornl.gov kernel: [1914649.846911] Lustre: atlas2-MDT0000-lwp-OST01d5: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3i6.ccs.ornl.gov kernel: [1914650.824135] Lustre: 25702:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880239aca000 x1471904509633868/t0(0) o400->atlas2-MDT0000-lwp-OST0165@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3i6.ccs.ornl.gov kernel: [1914650.825137] Lustre: 25704:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88049842d400 x1471904509633876/t0(0) o400->atlas2-MDT0000-lwp-OST0285@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3i6.ccs.ornl.gov kernel: [1914650.825150] Lustre: atlas2-MDT0000-lwp-OST0285: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3i6.ccs.ornl.gov kernel: [1914650.827105] Lustre: atlas2-MDT0000-lwp-OST0315: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3i6.ccs.ornl.gov kernel: [1914650.827108] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4a3.ccs.ornl.gov kernel: [1914667.882158] Lustre: 28204:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880748461c00 x1471904514688764/t0(0) o400->atlas2-MDT0000-lwp-OST028a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4a3.ccs.ornl.gov kernel: [1914667.882188] Lustre: atlas2-MDT0000-lwp-OST01fa: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4a3.ccs.ornl.gov kernel: [1914667.888558] Lustre: 28202:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880757b95000 x1471904514688756/t0(0) o400->atlas2-MDT0000-lwp-OST016a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4a3.ccs.ornl.gov kernel: [1914667.888563] Lustre: 28202:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4a3.ccs.ornl.gov kernel: [1914667.888571] Lustre: atlas2-MDT0000-lwp-OST016a: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4a3.ccs.ornl.gov kernel: [1914667.888573] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3e2.ccs.ornl.gov kernel: [1914651.335100] Lustre: 27201:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880297320800 x1471904513179056/t0(0) o400->atlas2-MDT0000-lwp-OST0141@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e2.ccs.ornl.gov kernel: [1914651.335119] Lustre: atlas2-MDT0000-lwp-OST01d1: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e2.ccs.ornl.gov kernel: [1914651.337091] Lustre: 27205:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88037f850400 x1471904513179072/t0(0) o400->atlas2-MDT0000-lwp-OST0381@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3e2.ccs.ornl.gov kernel: [1914651.337095] Lustre: 27205:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3e2.ccs.ornl.gov kernel: [1914651.337103] Lustre: atlas2-MDT0000-lwp-OST0381: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3e2.ccs.ornl.gov kernel: [1914651.337105] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4a1.ccs.ornl.gov kernel: [1914650.203081] Lustre: 25993:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4d1.ccs.ornl.gov kernel: [1914647.970123] Lustre: 22262:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff881009ab8000 x1471904516811588/t0(0) o400->atlas2-MDT0000-lwp-OST03c0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4d1.ccs.ornl.gov kernel: [1914647.970144] Lustre: atlas2-MDT0000-lwp-OST00f0: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4d1.ccs.ornl.gov kernel: [1914647.972129] Lustre: 22259:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880e5d2b0c00 x1471904516811584/t0(0) o400->atlas2-MDT0000-lwp-OST02a0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4d1.ccs.ornl.gov kernel: [1914647.972133] Lustre: 22259:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4d1.ccs.ornl.gov kernel: [1914647.972141] Lustre: atlas2-MDT0000-lwp-OST02a0: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4d1.ccs.ornl.gov kernel: [1914647.972143] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss4d1.ccs.ornl.gov kernel: [1914648.323687] Lustre: 22262:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3i3.ccs.ornl.gov kernel: [1914647.420646] Lustre: 22144:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880aa2edc000 x1471904518169128/t0(0) o400->atlas2-MDT0000-lwp-OST0162@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3a7.ccs.ornl.gov kernel: [1914650.102333] Lustre: 22092:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88061210e400 x1471904518655488/t0(0) o400->atlas2-MDT0000-lwp-OST0096@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3i3.ccs.ornl.gov kernel: [1914647.420658] Lustre: atlas2-MDT0000-lwp-OST00d2: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3i3.ccs.ornl.gov kernel: [1914647.423622] Lustre: 22145:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8809ef56a000 x1471904518169136/t0(0) o400->atlas2-MDT0000-lwp-OST0282@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3i3.ccs.ornl.gov kernel: [1914647.423626] Lustre: 22145:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3i3.ccs.ornl.gov kernel: [1914647.423634] Lustre: atlas2-MDT0000-lwp-OST0282: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3i3.ccs.ornl.gov kernel: [1914647.423636] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3a7.ccs.ornl.gov kernel: [1914650.102348] Lustre: atlas2-MDT0000-lwp-OST0366: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3a7.ccs.ornl.gov kernel: [1914650.106354] Lustre: 22080:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880579f8f400 x1471904518655504/t0(0) o400->atlas2-MDT0000-lwp-OST02d6@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3a7.ccs.ornl.gov kernel: [1914650.106358] Lustre: 22080:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3a7.ccs.ornl.gov kernel: [1914650.106366] Lustre: atlas2-MDT0000-lwp-OST02d6: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3d2.ccs.ornl.gov kernel: [1914653.824316] Lustre: 25988:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3d2.ccs.ornl.gov kernel: [1914653.824325] Lustre: atlas2-MDT0000-lwp-OST0259: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3d2.ccs.ornl.gov kernel: [1914653.824328] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3i6.ccs.ornl.gov kernel: [1914651.143194] Lustre: 25702:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3c3.ccs.ornl.gov kernel: [1914644.657064] Lustre: 22228:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8806e9dc8000 x1471904511051316/t0(0) o400->atlas2-MDT0000-lwp-OST0252@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3c3.ccs.ornl.gov kernel: [1914644.657081] Lustre: atlas2-MDT0000-lwp-OST0012: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3c3.ccs.ornl.gov kernel: [1914644.659050] Lustre: 22227:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880727828800 x1471904511051308/t0(0) o400->atlas2-MDT0000-lwp-OST0132@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3c3.ccs.ornl.gov kernel: [1914644.659054] Lustre: 22227:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3c3.ccs.ornl.gov kernel: [1914644.659058] Lustre: atlas2-MDT0000-lwp-OST00a2: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3c3.ccs.ornl.gov kernel: [1914644.659064] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3c3.ccs.ornl.gov kernel: [1914645.005512] Lustre: 22228:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3c4.ccs.ornl.gov kernel: [1914647.507369] Lustre: 25962:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880515d13400 x1471904511832792/t0(0) o400->atlas2-MDT0000-lwp-OST01c3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4b4.ccs.ornl.gov kernel: [1914650.057057] Lustre: 26020:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8802ae45c000 x1471904528871876/t0(0) o400->atlas2-MDT0000-lwp-OST00e3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4b4.ccs.ornl.gov kernel: [1914650.057061] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8805cb57d000 x1471904528871896/t0(0) o400->atlas2-MDT0000-lwp-OST03b3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4b4.ccs.ornl.gov kernel: [1914650.057066] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 256535 previous similar messages Jul 17 16:44:57 atlas-oss4b4.ccs.ornl.gov kernel: [1914650.057071] Lustre: atlas2-MDT0000-lwp-OST0053: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4b4.ccs.ornl.gov kernel: [1914650.057074] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss4b4.ccs.ornl.gov kernel: [1914650.059052] Lustre: atlas2-MDT0000-lwp-OST0323: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4b4.ccs.ornl.gov kernel: [1914650.059055] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss4b4.ccs.ornl.gov kernel: [1914650.429840] Lustre: 26020:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3g3.ccs.ornl.gov kernel: [1913249.359381] Lustre: 13965:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880d58d50000 x1471904516976640/t0(0) o400->atlas2-MDT0000-lwp-OST0392@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3g3.ccs.ornl.gov kernel: [1913249.359403] Lustre: atlas2-MDT0000-lwp-OST0302: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g3.ccs.ornl.gov kernel: [1913249.376413] Lustre: 13959:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880dea5b2c00 x1471904516976616/t0(0) o400->atlas2-MDT0000-lwp-OST0032@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3g3.ccs.ornl.gov kernel: [1913249.376417] Lustre: 13959:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3g3.ccs.ornl.gov kernel: [1913249.376426] Lustre: atlas2-MDT0000-lwp-OST0032: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3g3.ccs.ornl.gov kernel: [1913249.376428] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3b1.ccs.ornl.gov kernel: [1914658.625266] Lustre: 22326:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880212a3c800 x1471904516110608/t0(0) o400->atlas2-MDT0000-lwp-OST0098@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3b1.ccs.ornl.gov kernel: [1914658.625276] Lustre: atlas2-MDT0000-lwp-OST0128: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b1.ccs.ornl.gov kernel: [1914658.627255] Lustre: 22312:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88031d896c00 x1471904516110620/t0(0) o400->atlas2-MDT0000-lwp-OST0248@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3b1.ccs.ornl.gov kernel: [1914658.627260] Lustre: 22312:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3b1.ccs.ornl.gov kernel: [1914658.627269] Lustre: atlas2-MDT0000-lwp-OST0248: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b1.ccs.ornl.gov kernel: [1914658.627271] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3b1.ccs.ornl.gov kernel: [1914658.973516] Lustre: 22326:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3e6.ccs.ornl.gov kernel: [1914649.898039] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3i3.ccs.ornl.gov kernel: [1914647.771444] Lustre: 22144:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3a7.ccs.ornl.gov kernel: [1914650.428075] Lustre: 22092:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4a8.ccs.ornl.gov kernel: [1914658.371088] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4h3.ccs.ornl.gov kernel: [1914665.013155] Lustre: Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3d7.ccs.ornl.gov kernel: [1914652.865098] Lustre: 22225:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880d8871c800 x1471904512149320/t0(0) o400->atlas2-MDT0000-lwp-OST025e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3d7.ccs.ornl.gov kernel: [1914652.865105] Lustre: atlas2-MDT0000-lwp-OST013e: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3d7.ccs.ornl.gov kernel: [1914652.867123] Lustre: 22222:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880feaa06c00 x1471904512149304/t0(0) o400->atlas2-MDT0000-lwp-OST001e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3d7.ccs.ornl.gov kernel: [1914652.867127] Lustre: 22222:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3d7.ccs.ornl.gov kernel: [1914652.867135] Lustre: atlas2-MDT0000-lwp-OST001e: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3d7.ccs.ornl.gov kernel: [1914652.867137] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3f1.ccs.ornl.gov kernel: [1914655.834850] Lustre: 24211:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88086b476000 x1471904508988304/t0(0) o400->atlas2-MDT0000-lwp-OST0268@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3f1.ccs.ornl.gov kernel: [1914655.834861] Lustre: atlas2-MDT0000-lwp-OST0028: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3f1.ccs.ornl.gov kernel: [1914655.837831] Lustre: 24214:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880499d1cc00 x1471904508988308/t0(0) o400->atlas2-MDT0000-lwp-OST0388@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3f1.ccs.ornl.gov kernel: [1914655.837836] Lustre: 24214:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3f1.ccs.ornl.gov kernel: [1914655.837838] Lustre: atlas2-MDT0000-lwp-OST00b8: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3f1.ccs.ornl.gov kernel: [1914655.837844] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4f8.ccs.ornl.gov kernel: [1914651.255951] Lustre: 22227:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8805cff1b800 x1471904511805884/t0(0) o400->atlas2-MDT0000-lwp-OST0077@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4f8.ccs.ornl.gov kernel: [1914651.255961] Lustre: atlas2-MDT0000-lwp-OST0227: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4f8.ccs.ornl.gov kernel: [1914651.258944] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff88060dd6a400 x1471904511805908/t0(0) o400->atlas2-MDT0000-lwp-OST03d7@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4f8.ccs.ornl.gov kernel: [1914651.258948] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss4f8.ccs.ornl.gov kernel: [1914651.258955] Lustre: atlas2-MDT0000-lwp-OST03d7: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4f8.ccs.ornl.gov kernel: [1914651.258957] Lustre: Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss4d8.ccs.ornl.gov kernel: [1914665.000003] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8804157bc800 x1471904516222864/t0(0) o400->atlas2-MDT0000-lwp-OST02a7@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4d8.ccs.ornl.gov kernel: [1914665.000008] Lustre: 22124:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8804157bcc00 x1471904516222872/t0(0) o400->atlas2-MDT0000-lwp-OST03c7@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4d8.ccs.ornl.gov kernel: [1914665.000020] Lustre: atlas2-MDT0000-lwp-OST03c7: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4d8.ccs.ornl.gov kernel: [1914665.003983] Lustre: 22120:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8806be9e7400 x1471904516222856/t0(0) o400->atlas2-MDT0000-lwp-OST0187@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss4d8.ccs.ornl.gov kernel: [1914665.003994] Lustre: atlas2-MDT0000-lwp-OST0187: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss4d8.ccs.ornl.gov kernel: [1914665.397713] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3b7.ccs.ornl.gov kernel: [1914656.854165] Lustre: atlas2-MDT0000-lwp-OST012e: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b7.ccs.ornl.gov kernel: [1914657.004203] Lustre: 26035:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3b7.ccs.ornl.gov kernel: [1914657.004285] Lustre: 26049:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880126a6b800 x1471904519392516/t0(0) o400->atlas2-MDT0000-lwp-OST024e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3b7.ccs.ornl.gov kernel: [1914657.004289] Lustre: 26049:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3b7.ccs.ornl.gov kernel: [1914657.004297] Lustre: atlas2-MDT0000-lwp-OST024e: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b7.ccs.ornl.gov kernel: [1914657.004299] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3i1.ccs.ornl.gov kernel: [1914652.279842] Lustre: 22141:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8805dbbd7400 x1471904519755284/t0(0) o400->atlas2-MDT0000-lwp-OST03a0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3i1.ccs.ornl.gov kernel: [1914652.279846] Lustre: 22141:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3i1.ccs.ornl.gov kernel: [1914652.279854] Lustre: atlas2-MDT0000-lwp-OST03a0: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3i1.ccs.ornl.gov kernel: [1914652.279856] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3b8.ccs.ornl.gov kernel: [1914666.193231] Lustre: 22227:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880d205a2000 x1471904516406636/t0(0) o400->atlas2-MDT0000-lwp-OST009f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3b8.ccs.ornl.gov kernel: [1914666.193250] Lustre: atlas2-MDT0000-lwp-OST012f: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b8.ccs.ornl.gov kernel: [1914666.193254] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3b8.ccs.ornl.gov kernel: [1914666.196226] Lustre: 22230:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880fd3654c00 x1471904516406648/t0(0) o400->atlas2-MDT0000-lwp-OST024f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3b8.ccs.ornl.gov kernel: [1914666.196230] Lustre: 22230:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:44:57 atlas-oss3b8.ccs.ornl.gov kernel: [1914666.196238] Lustre: atlas2-MDT0000-lwp-OST024f: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b8.ccs.ornl.gov kernel: [1914666.196240] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3d5.ccs.ornl.gov kernel: [1914655.450427] Lustre: 22190:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff8801afe5e000 x1471904509641888/t0(0) o400->atlas2-MDT0000-lwp-OST00ac@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3d5.ccs.ornl.gov kernel: [1914655.450433] Lustre: 22193:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880453aa3000 x1471904509641900/t0(0) o400->atlas2-MDT0000-lwp-OST025c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3d5.ccs.ornl.gov kernel: [1914655.450449] Lustre: atlas2-MDT0000-lwp-OST025c: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3d5.ccs.ornl.gov kernel: [1914655.450452] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3d5.ccs.ornl.gov kernel: [1914655.451427] Lustre: 22180:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880837f3ac00 x1471904509641908/t0(0) o400->atlas2-MDT0000-lwp-OST037c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3d5.ccs.ornl.gov kernel: [1914655.451432] Lustre: 22180:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3d5.ccs.ornl.gov kernel: [1914655.451440] Lustre: atlas2-MDT0000-lwp-OST037c: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3d5.ccs.ornl.gov kernel: [1914655.451443] Lustre: Skipped 2 previous similar messages Jul 17 16:44:57 atlas-oss3b4.ccs.ornl.gov kernel: [1914663.696270] Lustre: 26925:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880daae97800 x1471904510548432/t0(0) o400->atlas2-MDT0000-lwp-OST02db@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3b4.ccs.ornl.gov kernel: [1914663.696278] Lustre: atlas2-MDT0000-lwp-OST012b: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b4.ccs.ornl.gov kernel: [1914663.698250] Lustre: 26924:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880b2f397c00 x1471904510548428/t0(0) o400->atlas2-MDT0000-lwp-OST024b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3b4.ccs.ornl.gov kernel: [1914663.698255] Lustre: 26924:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:44:57 atlas-oss3b4.ccs.ornl.gov kernel: [1914663.698262] Lustre: atlas2-MDT0000-lwp-OST024b: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3b4.ccs.ornl.gov kernel: [1914663.698264] Lustre: Skipped 3 previous similar messages Jul 17 16:44:57 atlas-oss3b4.ccs.ornl.gov kernel: [1914664.042136] Lustre: 26925:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3i2.ccs.ornl.gov kernel: [1914658.361385] Lustre: 25727:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629330/real 1405629330] req@ffff880e08d04000 x1471904514016140/t0(0) o400->atlas2-MDT0000-lwp-OST03a1@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629897 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:44:57 atlas-oss3i2.ccs.ornl.gov kernel: [1914658.361415] Lustre: atlas2-MDT0000-lwp-OST01f1: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3i2.ccs.ornl.gov kernel: [1914658.361419] Lustre: Skipped 1 previous similar message Jul 17 16:44:57 atlas-oss3i2.ccs.ornl.gov kernel: [1914658.525528] Lustre: 25727:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:44:57 atlas-oss3i2.ccs.ornl.gov kernel: [1914658.556016] Lustre: atlas2-MDT0000-lwp-OST03a1: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:44:57 atlas-oss3i2.ccs.ornl.gov kernel: [1914658.606904] Lustre: Skipped 4 previous similar messages Jul 17 16:45:43 atlas-oss3i5.ccs.ornl.gov kernel: [701006.075659] Lustre: 17781:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629376/real 1405629376] req@ffff8807482da400 x1473175721933200/t0(0) o400->atlas2-MDT0000-lwp-OST01f4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:45:43 atlas-oss3i5.ccs.ornl.gov kernel: [701006.076615] Lustre: atlas2-MDT0000-lwp-OST0044: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:45:43 atlas-oss3i5.ccs.ornl.gov kernel: [701006.077607] Lustre: 17767:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629376/real 1405629376] req@ffff88071c2d3400 x1473175721933208/t0(0) o400->atlas2-MDT0000-lwp-OST0284@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:45:43 atlas-oss3i5.ccs.ornl.gov kernel: [701006.077612] Lustre: 17767:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:45:43 atlas-oss3i5.ccs.ornl.gov kernel: [701006.077620] Lustre: atlas2-MDT0000-lwp-OST0314: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:45:43 atlas-oss3i5.ccs.ornl.gov kernel: [701006.077623] Lustre: Skipped 2 previous similar messages Jul 17 16:45:43 atlas-oss3i5.ccs.ornl.gov kernel: [701006.420205] Lustre: 17781:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:45:43 atlas-oss3a8.ccs.ornl.gov kernel: [701007.013005] Lustre: 17742:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629376/real 1405629376] req@ffff880510213c00 x1473175720194792/t0(0) o400->atlas2-MDT0000-lwp-OST02d7@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:45:43 atlas-oss3a8.ccs.ornl.gov kernel: [701007.013011] Lustre: atlas2-MDT0000-lwp-OST0247: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:45:43 atlas-oss3a8.ccs.ornl.gov kernel: [701007.015453] Lustre: 17740:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629376/real 1405629376] req@ffff88086efd0000 x1473175720194784/t0(0) o400->atlas2-MDT0000-lwp-OST0007@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:45:43 atlas-oss3a8.ccs.ornl.gov kernel: [701007.015457] Lustre: 17740:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:45:43 atlas-oss3a8.ccs.ornl.gov kernel: [701007.015464] Lustre: atlas2-MDT0000-lwp-OST0007: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:45:43 atlas-oss3a8.ccs.ornl.gov kernel: [701007.015466] Lustre: Skipped 2 previous similar messages Jul 17 16:45:43 atlas-oss3a8.ccs.ornl.gov kernel: [701007.366261] Lustre: 17742:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:45:43 atlas-oss3d4.ccs.ornl.gov kernel: [701006.689462] Lustre: 16396:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629376/real 1405629376] req@ffff880b5656c800 x1473175723812592/t0(0) o400->atlas2-MDT0000-lwp-OST037b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405629943 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:45:43 atlas-oss3d4.ccs.ornl.gov kernel: [701006.689482] Lustre: atlas2-MDT0000-lwp-OST02eb: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:45:43 atlas-oss3d4.ccs.ornl.gov kernel: [701006.833937] Lustre: 16396:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:45:43 atlas-oss3d4.ccs.ornl.gov kernel: [701006.872462] Lustre: atlas2-MDT0000-lwp-OST037b: Connection to atlas2-MDT0000 (at 10.36.226.74@o2ib) was lost; in progress operations using this service will wait for recovery to complete Jul 17 16:45:43 atlas-oss3d4.ccs.ornl.gov kernel: [701006.923388] Lustre: Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss3a6.ccs.ornl.gov kernel: [1914803.567598] Lustre: 25789:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8807a869ec00 x1471904512394532/t0(0) o400->atlas2-MDT0000-lwp-OST0005@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3c1.ccs.ornl.gov kernel: [1914803.094943] Lustre: 23691:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88028dd7f400 x1471904511392888/t0(0) o400->atlas2-MDT0000-lwp-OST02e0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3e7.ccs.ornl.gov kernel: [1914803.360067] Lustre: 25883:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880b80905c00 x1471904508195104/t0(0) o400->atlas2-MDT0000-lwp-OST00b6@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3e7.ccs.ornl.gov kernel: [1914803.461000] Lustre: 25883:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:47:26 atlas-oss4i8.ccs.ornl.gov kernel: [1914814.696892] Lustre: 26106:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880fe84ab000 x1471904520247032/t0(0) o400->atlas2-MDT0000-lwp-OST02cf@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3e1.ccs.ornl.gov kernel: [1914803.768252] Lustre: 23611:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88031fd69000 x1471904509694260/t0(0) o400->atlas2-MDT0000-lwp-OST02f0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4g8.ccs.ornl.gov kernel: [1914800.167576] Lustre: 25553:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8802d884d000 x1471904521746740/t0(0) o400->atlas2-MDT0000-lwp-OST03df@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4i4.ccs.ornl.gov kernel: [1914806.488339] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880c740fd400 x1471904516513340/t0(0) o400->atlas2-MDT0000-lwp-OST02cb@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4d3.ccs.ornl.gov kernel: [1914798.058011] Lustre: 22328:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880eb380ec00 x1471904516978164/t0(0) o400->atlas2-MDT0000-lwp-OST0182@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4g5.ccs.ornl.gov kernel: [1914801.708523] Lustre: 26061:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88069ab85800 x1471904510456640/t0(0) o400->atlas2-MDT0000-lwp-OST010c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4g5.ccs.ornl.gov kernel: [1914801.801410] Lustre: 26061:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss4h4.ccs.ornl.gov kernel: [1914821.010322] Lustre: 25839:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880867656c00 x1471904512334272/t0(0) o400->atlas2-MDT0000-lwp-OST0353@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4h4.ccs.ornl.gov kernel: [1914821.096507] Lustre: 25839:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss4g2.ccs.ornl.gov kernel: [1914812.369472] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88076a959400 x1471904509228892/t0(0) o400->atlas2-MDT0000-lwp-OST03d9@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4e4.ccs.ornl.gov kernel: [1914810.301562] Lustre: 22199:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8802c5dfe800 x1471904515374232/t0(0) o400->atlas2-MDT0000-lwp-OST021b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4e4.ccs.ornl.gov kernel: [1914810.394782] Lustre: 22199:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:47:26 atlas-oss4f2.ccs.ornl.gov kernel: [1914805.185398] Lustre: 25725:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8804b393b800 x1471904508910600/t0(0) o400->atlas2-MDT0000-lwp-OST03d1@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4f2.ccs.ornl.gov kernel: [1914805.269867] Lustre: 25725:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss4i2.ccs.ornl.gov kernel: [1914824.472216] Lustre: 25753:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880739247400 x1471904514634440/t0(0) o400->atlas2-MDT0000-lwp-OST0239@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3i7.ccs.ornl.gov kernel: [1914798.768324] Lustre: 24584:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880cfa3c5000 x1471904508389212/t0(0) o400->atlas2-MDT0000-lwp-OST03a6@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4d3.ccs.ornl.gov kernel: [1914798.151252] Lustre: 22328:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss4h7.ccs.ornl.gov kernel: [1914809.902927] Lustre: 26114:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88085e101800 x1471904512615056/t0(0) o400->atlas2-MDT0000-lwp-OST0236@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4i4.ccs.ornl.gov kernel: [1914806.573140] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss4h7.ccs.ornl.gov kernel: [1914809.990463] Lustre: 26114:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss4h6.ccs.ornl.gov kernel: [1914800.424174] Lustre: 25971:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88104ebfe800 x1471904515532020/t0(0) o400->atlas2-MDT0000-lwp-OST03e5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4d2.ccs.ornl.gov kernel: [1914806.719585] Lustre: 27444:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880747f9ac00 x1471904512220864/t0(0) o400->atlas2-MDT0000-lwp-OST00f1@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4h6.ccs.ornl.gov kernel: [1914800.512043] Lustre: 25971:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss4d2.ccs.ornl.gov kernel: [1914806.809610] Lustre: 27444:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss4a4.ccs.ornl.gov kernel: [1914803.177301] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88025ceb5400 x1471904509590996/t0(0) o400->atlas2-MDT0000-lwp-OST016b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4a4.ccs.ornl.gov kernel: [1914803.265795] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss4b3.ccs.ornl.gov kernel: [1914795.939914] Lustre: 22306:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88032e925c00 x1471904509180424/t0(0) o400->atlas2-MDT0000-lwp-OST0172@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4i7.ccs.ornl.gov kernel: [1914814.905588] Lustre: 25160:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88091857e400 x1471904511375012/t0(0) o400->atlas2-MDT0000-lwp-OST01ae@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4f7.ccs.ornl.gov kernel: [1914800.701185] Lustre: 25982:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880443d29800 x1471904510454960/t0(0) o400->atlas2-MDT0000-lwp-OST03d6@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4f7.ccs.ornl.gov kernel: [1914800.793945] Lustre: 25982:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss4a5.ccs.ornl.gov kernel: [1914820.594962] Lustre: 22131:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff881012eba000 x1471904519454040/t0(0) o400->atlas2-MDT0000-lwp-OST00dc@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4c8.ccs.ornl.gov kernel: [1914793.513067] Lustre: 22046:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880768634c00 x1471904512196828/t0(0) o400->atlas2-MDT0000-lwp-OST032f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4a5.ccs.ornl.gov kernel: [1914820.686064] Lustre: 22131:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:47:26 atlas-oss4g2.ccs.ornl.gov kernel: [1914812.459424] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss4b2.ccs.ornl.gov kernel: [1914824.444457] Lustre: 27473:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880d1e0d0c00 x1471904518025748/t0(0) o400->atlas2-MDT0000-lwp-OST0171@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4i2.ccs.ornl.gov kernel: [1914824.560126] Lustre: 25753:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss4c5.ccs.ornl.gov kernel: [1914797.385677] Lustre: 26669:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88064d154800 x1471904510171132/t0(0) o400->atlas2-MDT0000-lwp-OST029c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4c5.ccs.ornl.gov kernel: [1914797.487081] Lustre: 26669:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:47:26 atlas-oss4e7.ccs.ornl.gov kernel: [1914799.058092] Lustre: 26094:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880772e98000 x1471904522621404/t0(0) o400->atlas2-MDT0000-lwp-OST006e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4e7.ccs.ornl.gov kernel: [1914799.148848] Lustre: 26094:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss4h2.ccs.ornl.gov kernel: [1914822.863826] Lustre: 26001:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880748917000 x1471904508411032/t0(0) o400->atlas2-MDT0000-lwp-OST0081@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3i7.ccs.ornl.gov kernel: [1914798.859831] Lustre: 24584:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:47:26 atlas-oss3h8.ccs.ornl.gov kernel: [1914798.517274] Lustre: 22192:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88066f0d2400 x1471904514772732/t0(0) o400->atlas2-MDT0000-lwp-OST030f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3h8.ccs.ornl.gov kernel: [1914798.610050] Lustre: 22192:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss3d6.ccs.ornl.gov kernel: [1914816.038935] Lustre: 25618:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff881002a1e800 x1471904511183292/t0(0) o400->atlas2-MDT0000-lwp-OST02ed@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3d6.ccs.ornl.gov kernel: [1914816.125406] Lustre: 25618:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss4b3.ccs.ornl.gov kernel: [1914796.024646] Lustre: 22306:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss4a2.ccs.ornl.gov kernel: [1914816.297703] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88035b420c00 x1471904510895600/t0(0) o400->atlas2-MDT0000-lwp-OST0049@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4a2.ccs.ornl.gov kernel: [1914816.381555] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss4i7.ccs.ornl.gov kernel: [1914814.994285] Lustre: 25160:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss3e3.ccs.ornl.gov kernel: [1914798.366567] Lustre: 23070:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88056f0be000 x1471904510468344/t0(0) o400->atlas2-MDT0000-lwp-OST0022@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3e3.ccs.ornl.gov kernel: [1914798.455401] Lustre: 23070:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss4e2.ccs.ornl.gov kernel: [1914815.475152] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880b7169d400 x1471904514146248/t0(0) o400->atlas2-MDT0000-lwp-OST02a9@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4e2.ccs.ornl.gov kernel: [1914815.565573] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss4h2.ccs.ornl.gov kernel: [1914822.954469] Lustre: 26001:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss4d7.ccs.ornl.gov kernel: [1914802.383759] Lustre: 22278:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880fe644c000 x1471904514719896/t0(0) o400->atlas2-MDT0000-lwp-OST0336@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4d7.ccs.ornl.gov kernel: [1914802.469179] Lustre: 22278:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss4c8.ccs.ornl.gov kernel: [1914793.599002] Lustre: 22046:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:47:26 atlas-oss4b2.ccs.ornl.gov kernel: [1914824.532482] Lustre: 27473:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss4i3.ccs.ornl.gov kernel: [1914793.925905] Lustre: 25960:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8807b5583400 x1471904523159500/t0(0) o400->atlas2-MDT0000-lwp-OST03ea@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4i3.ccs.ornl.gov kernel: [1914794.014009] Lustre: 25960:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:47:26 atlas-oss3h5.ccs.ornl.gov kernel: [1914821.406355] Lustre: 24599:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8801c3a40400 x1471904507806504/t0(0) o400->atlas2-MDT0000-lwp-OST039c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3h5.ccs.ornl.gov kernel: [1914821.494499] Lustre: 24599:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss3c1.ccs.ornl.gov kernel: [1914803.835199] Lustre: 23687:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8803ed40c400 x1471904511392876/t0(0) o400->atlas2-MDT0000-lwp-OST01c0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3c1.ccs.ornl.gov kernel: [1914803.928158] Lustre: 23687:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss3f6.ccs.ornl.gov kernel: [801506.296082] Lustre: 79666:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880a94a78000 x1473076856558476/t0(0) o400->atlas2-MDT0000-lwp-OST002d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3f6.ccs.ornl.gov kernel: [801506.382849] Lustre: 79666:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss4e1.ccs.ornl.gov kernel: [1914799.579827] Lustre: 26053:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880d9ec71800 x1471904515144644/t0(0) o400->atlas2-MDT0000-lwp-OST0068@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4e1.ccs.ornl.gov kernel: [1914799.671958] Lustre: 26053:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss3h3.ccs.ornl.gov kernel: [1914797.317279] Lustre: 26632:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88047fc73000 x1471904512413516/t0(0) o400->atlas2-MDT0000-lwp-OST030a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3h3.ccs.ornl.gov kernel: [1914797.405981] Lustre: 26632:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:47:26 atlas-oss4c4.ccs.ornl.gov kernel: [1914813.589067] Lustre: 25799:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880d5a161400 x1471904510980952/t0(0) o400->atlas2-MDT0000-lwp-OST005b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4c4.ccs.ornl.gov kernel: [1914813.681086] Lustre: 25799:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss4c3.ccs.ornl.gov kernel: [1914798.520450] Lustre: 24975:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880be7e02c00 x1471904512245972/t0(0) o400->atlas2-MDT0000-lwp-OST00ea@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4c3.ccs.ornl.gov kernel: [1914798.605947] Lustre: 24975:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss3f7.ccs.ornl.gov kernel: [801132.798735] Lustre: 78670:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880737c7a000 x1473076863306084/t0(0) o400->atlas2-MDT0000-lwp-OST00be@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3f7.ccs.ornl.gov kernel: [801132.884093] Lustre: 78670:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss3h7.ccs.ornl.gov kernel: [1914808.445839] Lustre: 25097:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88077e636800 x1471904510332852/t0(0) o400->atlas2-MDT0000-lwp-OST030e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3h7.ccs.ornl.gov kernel: [1914808.536936] Lustre: 25097:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss3h2.ccs.ornl.gov kernel: [1914804.782986] Lustre: 20931:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880c6062f800 x1471904518760416/t0(0) o400->atlas2-MDT0000-lwp-OST0309@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3h2.ccs.ornl.gov kernel: [1914804.871232] Lustre: 20931:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss4c1.ccs.ornl.gov kernel: [1914807.236332] Lustre: 22139:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8805a25bc800 x1471904516097412/t0(0) o400->atlas2-MDT0000-lwp-OST0298@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3f4.ccs.ornl.gov kernel: [1914803.915488] Lustre: 22158:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880d72543c00 x1471904507778368/t0(0) o400->atlas2-MDT0000-lwp-OST002b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3f4.ccs.ornl.gov kernel: [1914804.005644] Lustre: 22158:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:47:26 atlas-oss3g8.ccs.ornl.gov kernel: [1913399.795672] Lustre: 13441:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8809b4455800 x1471904513602332/t0(0) o400->atlas2-MDT0000-lwp-OST0397@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3b3.ccs.ornl.gov kernel: [1914794.726723] Lustre: 25600:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88082f0fa800 x1471904513820332/t0(0) o400->atlas2-MDT0000-lwp-OST024a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3b3.ccs.ornl.gov kernel: [1914794.816660] Lustre: 25600:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:47:26 atlas-oss4b1.ccs.ornl.gov kernel: [1914806.249598] Lustre: 25864:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88023ef1ac00 x1471904507948984/t0(0) o400->atlas2-MDT0000-lwp-OST00e0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4b1.ccs.ornl.gov kernel: [1914806.335843] Lustre: 25864:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss3a5.ccs.ornl.gov kernel: [1914795.303251] Lustre: 22231:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88058fd69000 x1471904511738380/t0(0) o400->atlas2-MDT0000-lwp-OST0364@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3f8.ccs.ornl.gov kernel: [801504.166881] Lustre: 81096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8803d1176400 x1473076861320852/t0(0) o400->atlas2-MDT0000-lwp-OST026f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3f8.ccs.ornl.gov kernel: [801504.262384] Lustre: 81096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss4c1.ccs.ornl.gov kernel: [1914807.327862] Lustre: 22139:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss4f6.ccs.ornl.gov kernel: [1914796.553236] Lustre: 22294:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff881046e4e000 x1471904511708448/t0(0) o400->atlas2-MDT0000-lwp-OST0075@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4f6.ccs.ornl.gov kernel: [1914796.640028] Lustre: 22294:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss3f5.ccs.ornl.gov kernel: [801493.676836] Lustre: 79212:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880f8f936000 x1473076856406192/t0(0) o400->atlas2-MDT0000-lwp-OST014c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3f5.ccs.ornl.gov kernel: [801493.769990] Lustre: 79212:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss3g5.ccs.ornl.gov kernel: [1913403.368439] Lustre: 13561:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8802617d3000 x1471904518248988/t0(0) o400->atlas2-MDT0000-lwp-OST01e4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss3g5.ccs.ornl.gov kernel: [1913403.458630] Lustre: 13561:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss3g8.ccs.ornl.gov kernel: [1913399.887219] Lustre: 13441:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:26 atlas-oss4f5.ccs.ornl.gov kernel: [1914799.714337] Lustre: 22147:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880a5f6b8400 x1471904509390840/t0(0) o400->atlas2-MDT0000-lwp-OST0344@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4f5.ccs.ornl.gov kernel: [1914799.804307] Lustre: 22147:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:47:26 atlas-oss4c2.ccs.ornl.gov kernel: [1914805.730406] Lustre: 25984:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880755431000 x1471904508570576/t0(0) o400->atlas2-MDT0000-lwp-OST0329@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:26 atlas-oss4c2.ccs.ornl.gov kernel: [1914805.816067] Lustre: 25984:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:26 atlas-oss3a5.ccs.ornl.gov kernel: [1914795.389084] Lustre: 22231:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:47:26 atlas-oss3e6.ccs.ornl.gov kernel: [1914799.375034] Lustre: 22097:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880c04eb1c00 x1471904520534524/t0(0) o400->atlas2-MDT0000-lwp-OST0025@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4b7.ccs.ornl.gov kernel: [1914795.879629] Lustre: 23156:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8808b65e8400 x1471904509377180/t0(0) o400->atlas2-MDT0000-lwp-OST0056@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3i4.ccs.ornl.gov kernel: [1914807.687551] Lustre: 27280:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8805347df800 x1471904510470284/t0(0) o400->atlas2-MDT0000-lwp-OST0283@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3a1.ccs.ornl.gov kernel: [1914796.653635] Lustre: 22438:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880eb12e2400 x1471904525931672/t0(0) o400->atlas2-MDT0000-lwp-OST01b0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3b5.ccs.ornl.gov kernel: [1914814.828359] Lustre: 25968:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8802ad30f800 x1471904514716860/t0(0) o400->atlas2-MDT0000-lwp-OST009c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3b5.ccs.ornl.gov kernel: [1914814.918669] Lustre: 25968:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:27 atlas-oss3e6.ccs.ornl.gov kernel: [1914799.463470] Lustre: 22097:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4b6.ccs.ornl.gov kernel: [1914799.454892] Lustre: 26095:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88040cd82000 x1471904511213456/t0(0) o400->atlas2-MDT0000-lwp-OST0055@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4b6.ccs.ornl.gov kernel: [1914799.544798] Lustre: 26095:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4h8.ccs.ornl.gov kernel: [1914792.781911] Lustre: 24637:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880256b1f400 x1471904514267104/t0(0) o400->atlas2-MDT0000-lwp-OST02c7@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4h8.ccs.ornl.gov kernel: [1914792.870793] Lustre: 24637:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3g7.ccs.ornl.gov kernel: [1913399.399505] Lustre: 13452:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88051a601800 x1471904520422140/t0(0) o400->atlas2-MDT0000-lwp-OST0036@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3g7.ccs.ornl.gov kernel: [1913399.488364] Lustre: 13452:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3h4.ccs.ornl.gov kernel: [1914805.619706] Lustre: 25741:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880d59d92800 x1471904509245176/t0(0) o400->atlas2-MDT0000-lwp-OST039b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3h4.ccs.ornl.gov kernel: [1914805.710115] Lustre: 25741:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3h1.ccs.ornl.gov kernel: [1914790.557431] Lustre: 22214:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8805873d3400 x1471904517765808/t0(0) o400->atlas2-MDT0000-lwp-OST0308@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3h1.ccs.ornl.gov kernel: [1914790.650391] Lustre: 22214:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:47:27 atlas-oss3i4.ccs.ornl.gov kernel: [1914807.772933] Lustre: 27280:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4e8.ccs.ornl.gov kernel: [1914814.798731] Lustre: 24603:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88077342b400 x1471904518472708/t0(0) o400->atlas2-MDT0000-lwp-OST03cf@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4e8.ccs.ornl.gov kernel: [1914814.891677] Lustre: 24603:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:47:27 atlas-oss4b7.ccs.ornl.gov kernel: [1914795.964052] Lustre: 23156:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4b8.ccs.ornl.gov kernel: [1914803.701286] Lustre: 25800:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880ee3896800 x1471904512563812/t0(0) o400->atlas2-MDT0000-lwp-OST0297@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4b8.ccs.ornl.gov kernel: [1914803.804475] Lustre: 25800:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:47:27 atlas-oss3a1.ccs.ornl.gov kernel: [1914796.739560] Lustre: 22438:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4i6.ccs.ornl.gov kernel: [1914815.265946] Lustre: 25729:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8805e9736400 x1471904510597492/t0(0) o400->atlas2-MDT0000-lwp-OST035d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4i6.ccs.ornl.gov kernel: [1914815.351501] Lustre: 25729:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:47:27 atlas-oss4g3.ccs.ornl.gov kernel: [1914813.021498] Lustre: 25824:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880efbfbf800 x1471904517610128/t0(0) o400->atlas2-MDT0000-lwp-OST019a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4g3.ccs.ornl.gov kernel: [1914813.113773] Lustre: 25824:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4f1.ccs.ornl.gov kernel: [1914806.062530] Lustre: 25968:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8806e3526c00 x1471904513370952/t0(0) o400->atlas2-MDT0000-lwp-OST0070@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4f1.ccs.ornl.gov kernel: [1914806.150479] Lustre: 25968:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:47:27 atlas-oss4f4.ccs.ornl.gov kernel: [1914802.356404] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8804adca2000 x1471904516980156/t0(0) o400->atlas2-MDT0000-lwp-OST0193@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4f4.ccs.ornl.gov kernel: [1914802.441252] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:47:27 atlas-oss4a6.ccs.ornl.gov kernel: [1914803.983747] Lustre: 25605:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880656354000 x1471904512502684/t0(0) o400->atlas2-MDT0000-lwp-OST03ad@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4d4.ccs.ornl.gov kernel: [1914822.230598] Lustre: 27390:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880871cc6400 x1471904513679044/t0(0) o400->atlas2-MDT0000-lwp-OST0063@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4d4.ccs.ornl.gov kernel: [1914822.331984] Lustre: 27390:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4g4.ccs.ornl.gov kernel: [1914816.352127] Lustre: 25814:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880747f42000 x1471904515530584/t0(0) o400->atlas2-MDT0000-lwp-OST03db@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4g4.ccs.ornl.gov kernel: [1914816.445423] Lustre: 25814:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4b5.ccs.ornl.gov kernel: [1914819.477226] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8804dd8cb400 x1471904518365552/t0(0) o400->atlas2-MDT0000-lwp-OST00e4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4b5.ccs.ornl.gov kernel: [1914819.561774] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:27 atlas-oss3c7.ccs.ornl.gov kernel: [1914802.810497] Lustre: 25758:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8802094ee800 x1471904510248588/t0(0) o400->atlas2-MDT0000-lwp-OST0376@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3c7.ccs.ornl.gov kernel: [1914802.897317] Lustre: 25758:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:47:27 atlas-oss3d3.ccs.ornl.gov kernel: [1914808.758243] Lustre: 25547:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8802c52ad400 x1471904510125608/t0(0) o400->atlas2-MDT0000-lwp-OST001a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3d3.ccs.ornl.gov kernel: [1914808.848345] Lustre: 25547:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4c6.ccs.ornl.gov kernel: [1914802.853503] Lustre: 22191:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880dcc84cc00 x1471904517937132/t0(0) o400->atlas2-MDT0000-lwp-OST032d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4g6.ccs.ornl.gov kernel: [1914811.479948] Lustre: 24696:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff88013e6ec400 x1471904507898584/t0(0) o400->atlas2-MDT0000-lwp-OST010d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4g6.ccs.ornl.gov kernel: [1914811.479953] Lustre: 24702:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff8802f8a8f000 x1471904507898604/t0(0) o400->atlas2-MDT0000-lwp-OST03dd@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4g6.ccs.ornl.gov kernel: [1914811.479960] Lustre: 24702:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:47:27 atlas-oss4g6.ccs.ornl.gov kernel: [1914811.699529] Lustre: 24696:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:47:27 atlas-oss3g2.ccs.ornl.gov kernel: [1913398.576283] Lustre: 14006:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8802d0fe6800 x1471904509036036/t0(0) o400->atlas2-MDT0000-lwp-OST0391@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3e5.ccs.ornl.gov kernel: [1914803.198260] Lustre: 25941:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8805aefea000 x1471904510654704/t0(0) o400->atlas2-MDT0000-lwp-OST01d4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3e5.ccs.ornl.gov kernel: [1914803.283970] Lustre: 25941:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:27 atlas-oss3e4.ccs.ornl.gov kernel: [1914816.190001] Lustre: 22102:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880283abf800 x1471904508537928/t0(0) o400->atlas2-MDT0000-lwp-OST0263@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3e4.ccs.ornl.gov kernel: [1914816.280340] Lustre: 22102:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4a6.ccs.ornl.gov kernel: [1914804.070501] Lustre: 25605:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4f3.ccs.ornl.gov kernel: [1914796.672087] Lustre: 26013:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880108266800 x1471904513977056/t0(0) o400->atlas2-MDT0000-lwp-OST0342@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4d6.ccs.ornl.gov kernel: [1914802.038036] Lustre: 25554:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8806f81ef400 x1471904519610308/t0(0) o400->atlas2-MDT0000-lwp-OST00f5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4d6.ccs.ornl.gov kernel: [1914802.131013] Lustre: 25554:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:27 atlas-oss4g1.ccs.ornl.gov kernel: [1914799.039343] Lustre: 25822:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff88048d240000 x1471904508970304/t0(0) o400->atlas2-MDT0000-lwp-OST0348@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4g1.ccs.ornl.gov kernel: [1914799.127331] Lustre: 25822:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4a1.ccs.ornl.gov kernel: [1914799.913386] Lustre: 25997:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff881002c5e800 x1471904510345692/t0(0) o400->atlas2-MDT0000-lwp-OST0048@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3g1.ccs.ornl.gov kernel: [1913391.928327] Lustre: 13847:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880f08bd2000 x1471904510299536/t0(0) o400->atlas2-MDT0000-lwp-OST01e0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4a7.ccs.ornl.gov kernel: [1914813.616147] Lustre: 25802:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8801a1721400 x1471904510935920/t0(0) o400->atlas2-MDT0000-lwp-OST028e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4a7.ccs.ornl.gov kernel: [1914813.709558] Lustre: 25802:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:47:27 atlas-oss3a2.ccs.ornl.gov kernel: [1914801.499710] Lustre: 25627:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8809e82c6800 x1471904510524172/t0(0) o400->atlas2-MDT0000-lwp-OST0091@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4e3.ccs.ornl.gov kernel: [1914807.775439] Lustre: 27364:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880616ab4000 x1471904515523608/t0(0) o400->atlas2-MDT0000-lwp-OST006a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4e3.ccs.ornl.gov kernel: [1914807.868185] Lustre: 27364:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4g7.ccs.ornl.gov kernel: [1914824.130196] Lustre: 26058:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880d4463b000 x1471904508873476/t0(0) o400->atlas2-MDT0000-lwp-OST019e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4g7.ccs.ornl.gov kernel: [1914824.130201] Lustre: 26061:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629479/real 1405629479] req@ffff880a78c65000 x1471904508873488/t0(0) o400->atlas2-MDT0000-lwp-OST034e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630046 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4g7.ccs.ornl.gov kernel: [1914824.130206] Lustre: 26061:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:47:27 atlas-oss4g7.ccs.ornl.gov kernel: [1914824.348348] Lustre: 26058:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:47:27 atlas-oss4e6.ccs.ornl.gov kernel: [1914817.721435] Lustre: 25709:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880309b12000 x1471904508408008/t0(0) o400->atlas2-MDT0000-lwp-OST006d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4e6.ccs.ornl.gov kernel: [1914817.808615] Lustre: 25709:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:47:27 atlas-oss3g2.ccs.ornl.gov kernel: [1913398.667534] Lustre: 14006:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4d5.ccs.ornl.gov kernel: [1914814.483255] Lustre: 26101:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880eae299000 x1471904519136728/t0(0) o400->atlas2-MDT0000-lwp-OST02a4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4d5.ccs.ornl.gov kernel: [1914814.572087] Lustre: 26101:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:27 atlas-oss4a3.ccs.ornl.gov kernel: [1914817.938839] Lustre: 28195:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8802bd759c00 x1471904514688788/t0(0) o400->atlas2-MDT0000-lwp-OST016a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4a3.ccs.ornl.gov kernel: [1914818.024865] Lustre: 28195:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4c6.ccs.ornl.gov kernel: [1914802.942914] Lustre: 22191:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4f3.ccs.ornl.gov kernel: [1914796.771831] Lustre: 26013:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4a1.ccs.ornl.gov kernel: [1914799.998950] Lustre: 25997:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4i5.ccs.ornl.gov kernel: [1914809.938246] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff88081fb3fc00 x1471904512720300/t0(0) o400->atlas2-MDT0000-lwp-OST011c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4i5.ccs.ornl.gov kernel: [1914810.026513] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:47:27 atlas-oss3g1.ccs.ornl.gov kernel: [1913392.013918] Lustre: 13847:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4d1.ccs.ornl.gov kernel: [1914798.025343] Lustre: 22250:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880c6d17e000 x1471904516811604/t0(0) o400->atlas2-MDT0000-lwp-OST0180@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4c7.ccs.ornl.gov kernel: [1914801.859050] Lustre: 25662:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8806082afc00 x1471904514425512/t0(0) o400->atlas2-MDT0000-lwp-OST017e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4d1.ccs.ornl.gov kernel: [1914798.110441] Lustre: 22250:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:27 atlas-oss4c7.ccs.ornl.gov kernel: [1914801.949175] Lustre: 25662:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3c8.ccs.ornl.gov kernel: [1914807.035623] Lustre: 26038:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff88106133a000 x1471904507927940/t0(0) o400->atlas2-MDT0000-lwp-OST0137@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3c8.ccs.ornl.gov kernel: [1914807.128359] Lustre: 26038:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:47:27 atlas-oss4b4.ccs.ornl.gov kernel: [1914800.112079] Lustre: 26016:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8801ef4c6000 x1471904528871924/t0(0) o400->atlas2-MDT0000-lwp-OST0323@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4b4.ccs.ornl.gov kernel: [1914800.196691] Lustre: 26016:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:47:27 atlas-oss3e2.ccs.ornl.gov kernel: [1914801.390459] Lustre: 27210:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880859a6b000 x1471904513179092/t0(0) o400->atlas2-MDT0000-lwp-OST01d1@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3e2.ccs.ornl.gov kernel: [1914801.477145] Lustre: 27210:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:27 atlas-oss3i8.ccs.ornl.gov kernel: [1914815.888074] Lustre: 22174:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8805f0f57000 x1471904515355372/t0(0) o400->atlas2-MDT0000-lwp-OST01f7@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3i8.ccs.ornl.gov kernel: [1914815.974040] Lustre: 22174:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3a2.ccs.ornl.gov kernel: [1914801.587836] Lustre: 25627:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4h5.ccs.ornl.gov kernel: [1914804.499231] Lustre: 25680:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8806e38e0000 x1471904509313376/t0(0) o400->atlas2-MDT0000-lwp-OST03e4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4h5.ccs.ornl.gov kernel: [1914804.587280] Lustre: 25680:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3h6.ccs.ornl.gov kernel: [1914802.379724] Lustre: 25759:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880f5d75bc00 x1471904512216436/t0(0) o400->atlas2-MDT0000-lwp-OST039d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3h6.ccs.ornl.gov kernel: [1914802.464527] Lustre: 25759:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3i3.ccs.ornl.gov kernel: [1914797.475865] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880a105a2c00 x1471904518169176/t0(0) o400->atlas2-MDT0000-lwp-OST03a2@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3i3.ccs.ornl.gov kernel: [1914797.567366] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3a4.ccs.ornl.gov kernel: [1914791.998876] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8807b8d58c00 x1471904512538692/t0(0) o400->atlas2-MDT0000-lwp-OST02d3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3a4.ccs.ornl.gov kernel: [1914792.084098] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:27 atlas-oss3d2.ccs.ornl.gov kernel: [1914803.718888] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8805ec31f000 x1471904511214544/t0(0) o400->atlas2-MDT0000-lwp-OST0139@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3d2.ccs.ornl.gov kernel: [1914803.809580] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:27 atlas-oss3d1.ccs.ornl.gov kernel: [1914794.683696] Lustre: 25673:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8806d48e1800 x1471904515778216/t0(0) o400->atlas2-MDT0000-lwp-OST0258@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3d1.ccs.ornl.gov kernel: [1914794.769553] Lustre: 25673:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4d8.ccs.ornl.gov kernel: [1914815.056473] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8803b5464400 x1471904516222884/t0(0) o400->atlas2-MDT0000-lwp-OST00f7@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4d8.ccs.ornl.gov kernel: [1914815.145339] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3f2.ccs.ornl.gov kernel: [1914797.147430] Lustre: 25715:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880c5ec11800 x1471904513672412/t0(0) o400->atlas2-MDT0000-lwp-OST0029@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3f2.ccs.ornl.gov kernel: [1914797.237304] Lustre: 25715:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:47:27 atlas-oss3g6.ccs.ornl.gov kernel: [1913410.382262] Lustre: 13996:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8802a19ec400 x1471904523473716/t0(0) o400->atlas2-MDT0000-lwp-OST0035@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3g6.ccs.ornl.gov kernel: [1913410.468419] Lustre: 13996:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:27 atlas-oss3e8.ccs.ornl.gov kernel: [1914802.133176] Lustre: 25799:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8805413f1000 x1471904515139488/t0(0) o400->atlas2-MDT0000-lwp-OST0267@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3e8.ccs.ornl.gov kernel: [1914802.224493] Lustre: 25799:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:27 atlas-oss4a8.ccs.ornl.gov kernel: [1914808.059071] Lustre: 25959:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880ebd561c00 x1471904507750652/t0(0) o400->atlas2-MDT0000-lwp-OST004f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4a8.ccs.ornl.gov kernel: [1914808.148244] Lustre: 25959:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:27 atlas-oss3i6.ccs.ornl.gov kernel: [1914800.879280] Lustre: 25712:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8808184fe400 x1471904509633904/t0(0) o400->atlas2-MDT0000-lwp-OST01f5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3i6.ccs.ornl.gov kernel: [1914800.969472] Lustre: 25712:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:47:27 atlas-oss3c3.ccs.ornl.gov kernel: [1914794.712103] Lustre: 22221:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8804423e8800 x1471904511051348/t0(0) o400->atlas2-MDT0000-lwp-OST0252@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3c3.ccs.ornl.gov kernel: [1914794.801550] Lustre: 22221:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:47:27 atlas-oss3a3.ccs.ornl.gov kernel: [1914799.255510] Lustre: 27243:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff88104e77ec00 x1471904526379804/t0(0) o400->atlas2-MDT0000-lwp-OST0002@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3a3.ccs.ornl.gov kernel: [1914799.348533] Lustre: 27243:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3d5.ccs.ornl.gov kernel: [1914805.506194] Lustre: 22187:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8805a209e800 x1471904509641936/t0(0) o400->atlas2-MDT0000-lwp-OST02ec@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3d5.ccs.ornl.gov kernel: [1914805.592577] Lustre: 22187:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:47:27 atlas-oss4h1.ccs.ornl.gov kernel: [1914795.903368] Lustre: 25873:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880eca694000 x1471904511603016/t0(0) o400->atlas2-MDT0000-lwp-OST0110@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4h1.ccs.ornl.gov kernel: [1914795.993410] Lustre: 25873:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3c4.ccs.ornl.gov kernel: [1914797.315777] Lustre: 25957:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff88024b928400 x1471904511832832/t0(0) o400->atlas2-MDT0000-lwp-OST0373@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3c4.ccs.ornl.gov kernel: [1914797.402988] Lustre: 25957:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:27 atlas-oss3b7.ccs.ornl.gov kernel: [1914806.909783] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8803a0676400 x1471904519392544/t0(0) o400->atlas2-MDT0000-lwp-OST02de@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3b7.ccs.ornl.gov kernel: [1914807.001121] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:27 atlas-oss3a7.ccs.ornl.gov kernel: [1914800.157588] Lustre: 22085:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8807104cf800 x1471904518655524/t0(0) o400->atlas2-MDT0000-lwp-OST01b6@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3a7.ccs.ornl.gov kernel: [1914800.244504] Lustre: 22085:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3g3.ccs.ornl.gov kernel: [1913399.415162] Lustre: 13956:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880dc3f33400 x1471904516976668/t0(0) o400->atlas2-MDT0000-lwp-OST0302@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3g3.ccs.ornl.gov kernel: [1913399.501602] Lustre: 13956:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4e5.ccs.ornl.gov kernel: [1914803.113665] Lustre: 24656:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff88017c450000 x1471904515279096/t0(0) o400->atlas2-MDT0000-lwp-OST006c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4e5.ccs.ornl.gov kernel: [1914803.200887] Lustre: 24656:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3b1.ccs.ornl.gov kernel: [1914808.681324] Lustre: 22320:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff88030fad9c00 x1471904516110648/t0(0) o400->atlas2-MDT0000-lwp-OST01b8@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3b1.ccs.ornl.gov kernel: [1914808.770187] Lustre: 22320:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:47:27 atlas-oss4i1.ccs.ornl.gov kernel: [1914802.148847] Lustre: 25783:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880c2256dc00 x1471904513319620/t0(0) o400->atlas2-MDT0000-lwp-OST0118@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4i1.ccs.ornl.gov kernel: [1914802.240426] Lustre: 25783:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3b6.ccs.ornl.gov kernel: [1914800.266769] Lustre: 25956:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff88044dd48000 x1471904512275276/t0(0) o400->atlas2-MDT0000-lwp-OST012d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3b6.ccs.ornl.gov kernel: [1914800.353841] Lustre: 25956:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3f3.ccs.ornl.gov kernel: [1914815.555429] Lustre: 26015:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880b1edbe400 x1471904510687976/t0(0) o400->atlas2-MDT0000-lwp-OST002a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3f3.ccs.ornl.gov kernel: [1914815.648102] Lustre: 26015:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3b2.ccs.ornl.gov kernel: [1914802.041908] Lustre: 25282:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff881022c11400 x1471904511916908/t0(0) o400->atlas2-MDT0000-lwp-OST0249@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3b2.ccs.ornl.gov kernel: [1914802.126984] Lustre: 25282:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3i1.ccs.ornl.gov kernel: [1914802.191310] Lustre: 22128:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880371c32000 x1471904519755292/t0(0) o400->atlas2-MDT0000-lwp-OST0040@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3i1.ccs.ornl.gov kernel: [1914802.276235] Lustre: 22128:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3g4.ccs.ornl.gov kernel: [1913412.533083] Lustre: 13623:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff88096c022400 x1471904510791272/t0(0) o400->atlas2-MDT0000-lwp-OST0393@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3g4.ccs.ornl.gov kernel: [1913412.626318] Lustre: 13623:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3c2.ccs.ornl.gov kernel: [1914804.103532] Lustre: 25868:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8806bea27000 x1471904511647828/t0(0) o400->atlas2-MDT0000-lwp-OST00a1@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3c2.ccs.ornl.gov kernel: [1914804.188460] Lustre: 25868:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3d7.ccs.ornl.gov kernel: [1914802.920610] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880a67dff400 x1471904512149352/t0(0) o400->atlas2-MDT0000-lwp-OST025e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3c5.ccs.ornl.gov kernel: [1914808.665249] Lustre: 25811:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880f7f9bb000 x1471904512466024/t0(0) o400->atlas2-MDT0000-lwp-OST01c4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3c5.ccs.ornl.gov kernel: [1914808.754071] Lustre: 25811:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:47:27 atlas-oss3f1.ccs.ornl.gov kernel: [1914805.890600] Lustre: 24222:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff88056de2a000 x1471904508988340/t0(0) o400->atlas2-MDT0000-lwp-OST0388@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3f1.ccs.ornl.gov kernel: [1914805.982021] Lustre: 24222:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss4f8.ccs.ornl.gov kernel: [1914801.311328] Lustre: 22223:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8806d77f0000 x1471904511805932/t0(0) o400->atlas2-MDT0000-lwp-OST02b7@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4f8.ccs.ornl.gov kernel: [1914801.400261] Lustre: 22223:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3c6.ccs.ornl.gov kernel: [1914794.334393] Lustre: 24549:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8809fa3a9000 x1471904508186764/t0(0) o400->atlas2-MDT0000-lwp-OST0015@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3c6.ccs.ornl.gov kernel: [1914794.420912] Lustre: 24549:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3d7.ccs.ornl.gov kernel: [1914803.009805] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:47:27 atlas-oss3b8.ccs.ornl.gov kernel: [1914816.250771] Lustre: 22220:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880b7f51a000 x1471904516406672/t0(0) o400->atlas2-MDT0000-lwp-OST01bf@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3b8.ccs.ornl.gov kernel: [1914816.341740] Lustre: 22220:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:47:27 atlas-oss3d8.ccs.ornl.gov kernel: [1914808.536733] Lustre: 24575:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880e53d88000 x1471904508710576/t0(0) o400->atlas2-MDT0000-lwp-OST025f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3d8.ccs.ornl.gov kernel: [1914808.536738] Lustre: 24577:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880b3b146400 x1471904508710580/t0(0) o400->atlas2-MDT0000-lwp-OST02ef@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3d8.ccs.ornl.gov kernel: [1914808.536743] Lustre: 24577:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:47:27 atlas-oss3d8.ccs.ornl.gov kernel: [1914808.756682] Lustre: 24575:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:47:27 atlas-oss4h3.ccs.ornl.gov kernel: [1914814.700144] Lustre: 26069:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8805d776a800 x1471904509650744/t0(0) o400->atlas2-MDT0000-lwp-OST02c2@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4h3.ccs.ornl.gov kernel: [1914814.700150] Lustre: 26071:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff8806f0474400 x1471904509650748/t0(0) o400->atlas2-MDT0000-lwp-OST0232@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss4h3.ccs.ornl.gov kernel: [1914814.700155] Lustre: 26071:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:47:27 atlas-oss4h3.ccs.ornl.gov kernel: [1914814.930597] Lustre: 26069:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:47:27 atlas-oss3b4.ccs.ornl.gov kernel: [1914813.752553] Lustre: 26919:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880fdd3eb000 x1471904510548468/t0(0) o400->atlas2-MDT0000-lwp-OST036b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3b4.ccs.ornl.gov kernel: [1914813.838370] Lustre: 26919:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:47:27 atlas-oss3i2.ccs.ornl.gov kernel: [1914808.417330] Lustre: 25716:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629480/real 1405629480] req@ffff880f228b4400 x1471904514016160/t0(0) o400->atlas2-MDT0000-lwp-OST01f1@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630047 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:47:27 atlas-oss3i2.ccs.ornl.gov kernel: [1914808.503299] Lustre: 25716:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:48:13 atlas-oss3i5.ccs.ornl.gov kernel: [701156.131459] Lustre: 17775:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629526/real 1405629526] req@ffff88086b816400 x1473175721933236/t0(0) o400->atlas2-MDT0000-lwp-OST03a4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630093 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:48:13 atlas-oss3i5.ccs.ornl.gov kernel: [701156.216251] Lustre: 17775:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:48:13 atlas-oss3a8.ccs.ornl.gov kernel: [701157.068331] Lustre: 17745:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629526/real 1405629526] req@ffff88049b973400 x1473175720194804/t0(0) o400->atlas2-MDT0000-lwp-OST0097@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630093 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:48:13 atlas-oss3a8.ccs.ornl.gov kernel: [701157.153255] Lustre: 17745:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:48:13 atlas-oss3d4.ccs.ornl.gov kernel: [701156.745236] Lustre: 16400:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629526/real 1405629526] req@ffff880f00e6ac00 x1473175723812608/t0(0) o400->atlas2-MDT0000-lwp-OST025b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630093 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:48:13 atlas-oss3d4.ccs.ornl.gov kernel: [701156.745241] Lustre: 16399:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629526/real 1405629526] req@ffff880b54270800 x1473175723812600/t0(0) o400->atlas2-MDT0000-lwp-OST013b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630093 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:48:13 atlas-oss3d4.ccs.ornl.gov kernel: [701156.745245] Lustre: 16399:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:48:13 atlas-oss3d4.ccs.ornl.gov kernel: [701156.959870] Lustre: 16400:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:49:36 atlas-oss4b2.ccs.ornl.gov kernel: [1914953.761569] Lustre: 27461:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880b0f772800 x1471904518025852/t0(0) o38->atlas2-MDT0000-lwp-OST0051@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4b2.ccs.ornl.gov kernel: [1914953.851888] Lustre: 27461:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss3e3.ccs.ornl.gov kernel: [1914927.662107] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880277b70800 x1471904510468456/t0(0) o38->atlas2-MDT0000-lwp-OST0382@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4i3.ccs.ornl.gov kernel: [1914923.229073] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88106bac8000 x1471904523159588/t0(0) o38->atlas2-MDT0000-lwp-OST03ea@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4i3.ccs.ornl.gov kernel: [1914923.321616] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss4d7.ccs.ornl.gov kernel: [1914931.695878] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880a394e5000 x1471904514719988/t0(0) o38->atlas2-MDT0000-lwp-OST03c6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3h5.ccs.ornl.gov kernel: [1914950.702460] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8802b88dc800 x1471904507806592/t0(0) o38->atlas2-MDT0000-lwp-OST027c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3e3.ccs.ornl.gov kernel: [1914927.753690] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss4i4.ccs.ornl.gov kernel: [1914935.908637] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8809b9487000 x1471904516513420/t0(0) o38->atlas2-MDT0000-lwp-OST023b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3f6.ccs.ornl.gov kernel: [801635.590228] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880c742a1c00 x1473076856558584/t0(0) o38->atlas2-MDT0000-lwp-OST02fd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3c1.ccs.ornl.gov kernel: [1914933.143214] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880515218800 x1471904511392972/t0(0) o38->atlas2-MDT0000-lwp-OST00a0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3i7.ccs.ornl.gov kernel: [1914928.166074] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880a9da06c00 x1471904508389292/t0(0) o38->atlas2-MDT0000-lwp-OST0046@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4f2.ccs.ornl.gov kernel: [1914934.586487] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880b4daf4400 x1471904508910688/t0(0) o38->atlas2-MDT0000-lwp-OST0071@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4f2.ccs.ornl.gov kernel: [1914934.678883] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss4d3.ccs.ornl.gov kernel: [1914927.497705] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8809c4c14000 x1471904516978264/t0(0) o38->atlas2-MDT0000-lwp-OST02a2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4d3.ccs.ornl.gov kernel: [1914927.589721] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss4c4.ccs.ornl.gov kernel: [1914942.888647] Lustre: 25786:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880b01a0e000 x1471904510981060/t0(0) o38->atlas2-MDT0000-lwp-OST029b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4h4.ccs.ornl.gov kernel: [1914950.435318] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88084c0e4c00 x1471904512334360/t0(0) o38->atlas2-MDT0000-lwp-OST0233@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3h5.ccs.ornl.gov kernel: [1914950.794448] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss4h4.ccs.ornl.gov kernel: [1914950.525566] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss4c5.ccs.ornl.gov kernel: [1914926.782200] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8805a55e8c00 x1471904510171228/t0(0) o38->atlas2-MDT0000-lwp-OST032c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4c3.ccs.ornl.gov kernel: [1914927.820995] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880c0980ec00 x1471904512246076/t0(0) o38->atlas2-MDT0000-lwp-OST005a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4c3.ccs.ornl.gov kernel: [1914927.913972] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss3g8.ccs.ornl.gov kernel: [1913529.094596] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880afdb80400 x1471904513602412/t0(0) o38->atlas2-MDT0000-lwp-OST0037@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3g8.ccs.ornl.gov kernel: [1913529.185408] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss3h8.ccs.ornl.gov kernel: [1914927.916967] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8806a76d7c00 x1471904514772816/t0(0) o38->atlas2-MDT0000-lwp-OST00cf@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3h8.ccs.ornl.gov kernel: [1914928.008010] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss3d6.ccs.ornl.gov kernel: [1914945.431710] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880c70e52400 x1471904511183376/t0(0) o38->atlas2-MDT0000-lwp-OST025d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3f4.ccs.ornl.gov kernel: [1914933.222295] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880f0bae8000 x1471904507778468/t0(0) o38->atlas2-MDT0000-lwp-OST014b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4a4.ccs.ornl.gov kernel: [1914932.606199] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880303f97000 x1471904509591088/t0(0) o38->atlas2-MDT0000-lwp-OST016b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4a4.ccs.ornl.gov kernel: [1914932.694036] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss4a5.ccs.ornl.gov kernel: [1914950.013325] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880ade08b800 x1471904519454140/t0(0) o38->atlas2-MDT0000-lwp-OST03ac@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4c5.ccs.ornl.gov kernel: [1914926.875182] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:49:36 atlas-oss3e1.ccs.ornl.gov kernel: [1914933.817551] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8808718d6000 x1471904509694348/t0(0) o38->atlas2-MDT0000-lwp-OST0380@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4c8.ccs.ornl.gov kernel: [1914922.935624] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629897] req@ffff880d54c51000 x1471904512196908/t0(0) o38->atlas2-MDT0000-lwp-OST005f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4e7.ccs.ornl.gov kernel: [1914928.485553] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8805616b8800 x1471904522621516/t0(0) o38->atlas2-MDT0000-lwp-OST03ce@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3f5.ccs.ornl.gov kernel: [801622.978086] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88096782ec00 x1473076856406288/t0(0) o38->atlas2-MDT0000-lwp-OST02fc@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4a2.ccs.ornl.gov kernel: [1914945.698395] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8801f20a2000 x1471904510895712/t0(0) o38->atlas2-MDT0000-lwp-OST0049@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3f8.ccs.ornl.gov kernel: [801633.478602] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880ed7d0e800 x1473076861320936/t0(0) o38->atlas2-MDT0000-lwp-OST01df@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4c2.ccs.ornl.gov kernel: [1914935.023262] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880987276800 x1471904508570664/t0(0) o38->atlas2-MDT0000-lwp-OST0329@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3g5.ccs.ornl.gov kernel: [1913532.674567] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8803166c7800 x1471904518249088/t0(0) o38->atlas2-MDT0000-lwp-OST0154@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3e1.ccs.ornl.gov kernel: [1914933.908024] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:36 atlas-oss3f5.ccs.ornl.gov kernel: [801623.069149] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss4e1.ccs.ornl.gov kernel: [1914928.978270] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880dc8c09800 x1471904515144744/t0(0) o38->atlas2-MDT0000-lwp-OST0218@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4g8.ccs.ornl.gov kernel: [1914930.215596] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880565448c00 x1471904521746824/t0(0) o38->atlas2-MDT0000-lwp-OST022f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4g8.ccs.ornl.gov kernel: [1914930.306966] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:36 atlas-oss3h3.ccs.ornl.gov kernel: [1914926.721958] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880453152000 x1471904512413608/t0(0) o38->atlas2-MDT0000-lwp-OST039a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4d4.ccs.ornl.gov kernel: [1914951.459680] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88063e4adc00 x1471904513679156/t0(0) o38->atlas2-MDT0000-lwp-OST02a3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3a1.ccs.ornl.gov kernel: [1914925.954068] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880bd79c4400 x1471904525931764/t0(0) o38->atlas2-MDT0000-lwp-OST01b0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3h7.ccs.ornl.gov kernel: [1914937.847069] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88018a581c00 x1471904510332936/t0(0) o38->atlas2-MDT0000-lwp-OST030e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4e2.ccs.ornl.gov kernel: [1914944.931939] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880c5d7ab400 x1471904514146344/t0(0) o38->atlas2-MDT0000-lwp-OST00f9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4h8.ccs.ornl.gov kernel: [1914922.077418] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88082f8e6400 x1471904514267196/t0(0) o38->atlas2-MDT0000-lwp-OST0087@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3f7.ccs.ornl.gov kernel: [801262.192033] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880735573c00 x1473076863306192/t0(0) o38->atlas2-MDT0000-lwp-OST014e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4c1.ccs.ornl.gov kernel: [1914936.634613] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8801ebdb5000 x1471904516097504/t0(0) o38->atlas2-MDT0000-lwp-OST0178@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4b6.ccs.ornl.gov kernel: [1914928.761270] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88042720a800 x1471904511213560/t0(0) o38->atlas2-MDT0000-lwp-OST0325@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3g7.ccs.ornl.gov kernel: [1913528.705392] Lustre: 13449:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880273741800 x1471904520422252/t0(0) o38->atlas2-MDT0000-lwp-OST0276@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3b3.ccs.ornl.gov kernel: [1914924.124087] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8806f0b8f400 x1471904513820424/t0(0) o38->atlas2-MDT0000-lwp-OST024a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4b1.ccs.ornl.gov kernel: [1914935.642651] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88082d3e5800 x1471904507949084/t0(0) o38->atlas2-MDT0000-lwp-OST0290@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4e2.ccs.ornl.gov kernel: [1914945.025278] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss3f7.ccs.ornl.gov kernel: [801262.292910] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss4g3.ccs.ornl.gov kernel: [1914942.321885] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880fbbee5c00 x1471904517610232/t0(0) o38->atlas2-MDT0000-lwp-OST03da@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3a5.ccs.ornl.gov kernel: [1914924.715910] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88034da0a000 x1471904511738464/t0(0) o38->atlas2-MDT0000-lwp-OST0364@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3a6.ccs.ornl.gov kernel: [1914933.816895] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88010335ac00 x1471904512394640/t0(0) o38->atlas2-MDT0000-lwp-OST0245@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3a6.ccs.ornl.gov kernel: [1914933.907687] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:36 atlas-oss4f5.ccs.ornl.gov kernel: [1914929.110297] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629897] req@ffff880bdc579800 x1471904509390924/t0(0) o38->atlas2-MDT0000-lwp-OST02b4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3a5.ccs.ornl.gov kernel: [1914924.807857] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss3b5.ccs.ornl.gov kernel: [1914944.226023] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88063e4e4c00 x1471904514716964/t0(0) o38->atlas2-MDT0000-lwp-OST036c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3b5.ccs.ornl.gov kernel: [1914944.317480] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss3i4.ccs.ornl.gov kernel: [1914937.089624] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8803e420d000 x1471904510470372/t0(0) o38->atlas2-MDT0000-lwp-OST0283@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4f6.ccs.ornl.gov kernel: [1914925.996968] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8808dc606400 x1471904511708552/t0(0) o38->atlas2-MDT0000-lwp-OST0225@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4g7.ccs.ornl.gov kernel: [1914953.527262] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8810240c1400 x1471904508873576/t0(0) o38->atlas2-MDT0000-lwp-OST02be@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4g6.ccs.ornl.gov kernel: [1914940.906649] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88053f525000 x1471904507898672/t0(0) o38->atlas2-MDT0000-lwp-OST007d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4g6.ccs.ornl.gov kernel: [1914940.999365] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss3h2.ccs.ornl.gov kernel: [1914934.279039] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629897] req@ffff8808ef710c00 x1471904518760500/t0(0) o38->atlas2-MDT0000-lwp-OST0399@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3h2.ccs.ornl.gov kernel: [1914934.369928] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss4b7.ccs.ornl.gov kernel: [1914925.301331] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880af4aba800 x1471904509377292/t0(0) o38->atlas2-MDT0000-lwp-OST03b6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3h4.ccs.ornl.gov kernel: [1914935.016778] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880fcabd5c00 x1471904509245256/t0(0) o38->atlas2-MDT0000-lwp-OST039b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4b8.ccs.ornl.gov kernel: [1914933.101244] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880c6034f000 x1471904512563908/t0(0) o38->atlas2-MDT0000-lwp-OST0057@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3e7.ccs.ornl.gov kernel: [1914933.647318] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880d3b540000 x1471904508195196/t0(0) o38->atlas2-MDT0000-lwp-OST0266@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3e7.ccs.ornl.gov kernel: [1914933.739659] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:36 atlas-oss3h1.ccs.ornl.gov kernel: [1914919.987448] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88040b43e400 x1471904517765896/t0(0) o38->atlas2-MDT0000-lwp-OST0278@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4e8.ccs.ornl.gov kernel: [1914944.230221] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88070cb59000 x1471904518472784/t0(0) o38->atlas2-MDT0000-lwp-OST02af@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4f1.ccs.ornl.gov kernel: [1914935.458575] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880159b61800 x1471904513371056/t0(0) o38->atlas2-MDT0000-lwp-OST0070@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4f1.ccs.ornl.gov kernel: [1914935.548742] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:36 atlas-oss4i8.ccs.ornl.gov kernel: [1914944.976691] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880cf209c800 x1471904520247108/t0(0) o38->atlas2-MDT0000-lwp-OST008f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4i8.ccs.ornl.gov kernel: [1914945.069372] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:36 atlas-oss4i6.ccs.ornl.gov kernel: [1914944.697611] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8803b5a3e800 x1471904510597560/t0(0) o38->atlas2-MDT0000-lwp-OST023d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss4f4.ccs.ornl.gov kernel: [1914931.788276] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88080a665400 x1471904516980252/t0(0) o38->atlas2-MDT0000-lwp-OST0193@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:36 atlas-oss3e6.ccs.ornl.gov kernel: [1914929.019759] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629897] req@ffff880aec784800 x1471904520534628/t0(0) o38->atlas2-MDT0000-lwp-OST00b5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4g5.ccs.ornl.gov kernel: [1914932.008453] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8806f88df000 x1471904510456752/t0(0) o38->atlas2-MDT0000-lwp-OST034c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4g5.ccs.ornl.gov kernel: [1914932.100227] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss4e4.ccs.ornl.gov kernel: [1914940.592334] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8805a7c9a800 x1471904515374332/t0(0) o38->atlas2-MDT0000-lwp-OST021b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4g2.ccs.ornl.gov kernel: [1914942.677307] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88068c606c00 x1471904509228984/t0(0) o38->atlas2-MDT0000-lwp-OST0349@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4h7.ccs.ornl.gov kernel: [1914940.199373] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8804513d3000 x1471904512615160/t0(0) o38->atlas2-MDT0000-lwp-OST0356@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4h7.ccs.ornl.gov kernel: [1914940.289736] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss4d2.ccs.ornl.gov kernel: [1914937.017924] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880560045c00 x1471904512220972/t0(0) o38->atlas2-MDT0000-lwp-OST0331@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4d2.ccs.ornl.gov kernel: [1914937.108607] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss4i7.ccs.ornl.gov kernel: [1914945.203718] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880deb722000 x1471904511375120/t0(0) o38->atlas2-MDT0000-lwp-OST02ce@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4i7.ccs.ornl.gov kernel: [1914945.293765] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss4h6.ccs.ornl.gov kernel: [1914930.727060] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880ffef5d000 x1471904515532112/t0(0) o38->atlas2-MDT0000-lwp-OST0085@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4h6.ccs.ornl.gov kernel: [1914930.820462] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss4i2.ccs.ornl.gov kernel: [1914954.787901] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff8807185b2c00 x1471904514634544/t0(0) o38->atlas2-MDT0000-lwp-OST03e9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4i2.ccs.ornl.gov kernel: [1914954.880068] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss4f7.ccs.ornl.gov kernel: [1914931.009276] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff880353f0a000 x1471904510455056/t0(0) o38->atlas2-MDT0000-lwp-OST0196@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4h2.ccs.ornl.gov kernel: [1914953.163396] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88041c787400 x1471904508411148/t0(0) o38->atlas2-MDT0000-lwp-OST0081@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4b3.ccs.ornl.gov kernel: [1914926.240650] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629896/real 1405629896] req@ffff88082512fc00 x1471904509180532/t0(0) o38->atlas2-MDT0000-lwp-OST0292@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630176 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4f7.ccs.ornl.gov kernel: [1914931.102454] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss4d7.ccs.ornl.gov kernel: [1914932.788283] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880ee546e800 x1471904514719992/t0(0) o38->atlas2-MDT0000-lwp-OST0186@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4d7.ccs.ornl.gov kernel: [1914932.879035] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss3e3.ccs.ornl.gov kernel: [1914928.783532] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88070db5c400 x1471904510468460/t0(0) o38->atlas2-MDT0000-lwp-OST0142@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3f6.ccs.ornl.gov kernel: [801636.681649] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880914115c00 x1473076856558592/t0(0) o38->atlas2-MDT0000-lwp-OST002d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3f6.ccs.ornl.gov kernel: [801636.771941] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss4i3.ccs.ornl.gov kernel: [1914924.352495] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880b4224f800 x1471904523159592/t0(0) o38->atlas2-MDT0000-lwp-OST023a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4i3.ccs.ornl.gov kernel: [1914924.442206] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss3h5.ccs.ornl.gov kernel: [1914951.824902] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8808705f4000 x1471904507806596/t0(0) o38->atlas2-MDT0000-lwp-OST039c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3e3.ccs.ornl.gov kernel: [1914928.873987] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss4c3.ccs.ornl.gov kernel: [1914928.944425] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880d85de8000 x1471904512246084/t0(0) o38->atlas2-MDT0000-lwp-OST00ea@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4c3.ccs.ornl.gov kernel: [1914929.034803] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss3h5.ccs.ornl.gov kernel: [1914951.915184] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss3f8.ccs.ornl.gov kernel: [801634.571073] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880c88928000 x1473076861320948/t0(0) o38->atlas2-MDT0000-lwp-OST02ff@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3f8.ccs.ornl.gov kernel: [801634.660959] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss3g8.ccs.ornl.gov kernel: [1913530.216031] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880ea88d0000 x1471904513602424/t0(0) o38->atlas2-MDT0000-lwp-OST0157@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3g8.ccs.ornl.gov kernel: [1913530.315596] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss4c2.ccs.ornl.gov kernel: [1914936.114653] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8809f1ea7800 x1471904508570672/t0(0) o38->atlas2-MDT0000-lwp-OST00e9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4c2.ccs.ornl.gov kernel: [1914936.204797] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss3g5.ccs.ornl.gov kernel: [1913533.767003] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8805f7094400 x1471904518249092/t0(0) o38->atlas2-MDT0000-lwp-OST0034@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3g5.ccs.ornl.gov kernel: [1913533.857908] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss3f5.ccs.ornl.gov kernel: [801624.099528] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880fee8de400 x1473076856406300/t0(0) o38->atlas2-MDT0000-lwp-OST026c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3a1.ccs.ornl.gov kernel: [1914927.047483] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff881010a97000 x1471904525931768/t0(0) o38->atlas2-MDT0000-lwp-OST0120@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3a1.ccs.ornl.gov kernel: [1914927.137781] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss3f5.ccs.ornl.gov kernel: [801624.189184] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss4h8.ccs.ornl.gov kernel: [1914923.168812] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8804f88b8800 x1471904514267208/t0(0) o38->atlas2-MDT0000-lwp-OST03e7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4h8.ccs.ornl.gov kernel: [1914923.259644] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss4g4.ccs.ornl.gov kernel: [1914946.673254] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8801c0705c00 x1471904515530676/t0(0) o38->atlas2-MDT0000-lwp-OST019b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4b6.ccs.ornl.gov kernel: [1914929.852678] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8802d42fc000 x1471904511213572/t0(0) o38->atlas2-MDT0000-lwp-OST00e5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4b6.ccs.ornl.gov kernel: [1914929.943221] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss4f3.ccs.ornl.gov kernel: [1914926.968693] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8806be9c1400 x1471904513977152/t0(0) o38->atlas2-MDT0000-lwp-OST0222@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3g1.ccs.ornl.gov kernel: [1913522.221076] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8801fd2b7c00 x1471904510299636/t0(0) o38->atlas2-MDT0000-lwp-OST0030@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3g2.ccs.ornl.gov kernel: [1913528.895524] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88040398e000 x1471904509036128/t0(0) o38->atlas2-MDT0000-lwp-OST01e1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3a2.ccs.ornl.gov kernel: [1914931.794847] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880f8d7b7000 x1471904510524284/t0(0) o38->atlas2-MDT0000-lwp-OST02d1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4g3.ccs.ornl.gov kernel: [1914943.413263] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880a37bb7c00 x1471904517610236/t0(0) o38->atlas2-MDT0000-lwp-OST019a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4e3.ccs.ornl.gov kernel: [1914938.095829] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88046e731800 x1471904515523724/t0(0) o38->atlas2-MDT0000-lwp-OST033a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4d5.ccs.ornl.gov kernel: [1914944.788075] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8803bc5a2800 x1471904519136828/t0(0) o38->atlas2-MDT0000-lwp-OST03c4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3i8.ccs.ornl.gov kernel: [1914946.182212] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88021b90a800 x1471904515355476/t0(0) o38->atlas2-MDT0000-lwp-OST0287@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4i5.ccs.ornl.gov kernel: [1914940.242867] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88070d95a400 x1471904512720412/t0(0) o38->atlas2-MDT0000-lwp-OST03ec@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3d8.ccs.ornl.gov kernel: [1914938.854512] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880af9848000 x1471904508710676/t0(0) o38->atlas2-MDT0000-lwp-OST025f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4h5.ccs.ornl.gov kernel: [1914934.804846] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880394f5e400 x1471904509313468/t0(0) o38->atlas2-MDT0000-lwp-OST0114@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4g3.ccs.ornl.gov kernel: [1914943.503298] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss3a4.ccs.ornl.gov kernel: [1914922.299443] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8803cd3c4c00 x1471904512538788/t0(0) o38->atlas2-MDT0000-lwp-OST0363@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3i4.ccs.ornl.gov kernel: [1914938.182038] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88025d88dc00 x1471904510470384/t0(0) o38->atlas2-MDT0000-lwp-OST01f3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4d5.ccs.ornl.gov kernel: [1914944.881203] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss3f2.ccs.ornl.gov kernel: [1914927.444280] Lustre: 25710:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880ec902ec00 x1471904513672524/t0(0) o38->atlas2-MDT0000-lwp-OST00b9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4c7.ccs.ornl.gov kernel: [1914932.185202] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8803e677b400 x1471904514425620/t0(0) o38->atlas2-MDT0000-lwp-OST017e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4b5.ccs.ornl.gov kernel: [1914949.870674] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8807ea79c400 x1471904518365664/t0(0) o38->atlas2-MDT0000-lwp-OST0294@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4b5.ccs.ornl.gov kernel: [1914949.971168] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss3d1.ccs.ornl.gov kernel: [1914924.995440] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8804de44f000 x1471904515778316/t0(0) o38->atlas2-MDT0000-lwp-OST02e8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4h1.ccs.ornl.gov kernel: [1914926.210066] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880d7f867000 x1471904511603128/t0(0) o38->atlas2-MDT0000-lwp-OST01a0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3b6.ccs.ornl.gov kernel: [1914930.560772] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88075928c000 x1471904512275384/t0(0) o38->atlas2-MDT0000-lwp-OST009d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3e8.ccs.ornl.gov kernel: [1914932.451169] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88049fd0c800 x1471904515139588/t0(0) o38->atlas2-MDT0000-lwp-OST02f7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3e8.ccs.ornl.gov kernel: [1914932.543296] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss4e5.ccs.ornl.gov kernel: [1914933.417119] Lustre: 24644:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8808146b9000 x1471904515279208/t0(0) o38->atlas2-MDT0000-lwp-OST02ac@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4e5.ccs.ornl.gov kernel: [1914933.509857] Lustre: 24644:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss4i5.ccs.ornl.gov kernel: [1914940.335740] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:49:37 atlas-oss3b2.ccs.ornl.gov kernel: [1914932.335083] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880a61d9b000 x1471904511917008/t0(0) o38->atlas2-MDT0000-lwp-OST0009@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3a3.ccs.ornl.gov kernel: [1914929.576309] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff881008a59800 x1471904526379920/t0(0) o38->atlas2-MDT0000-lwp-OST0122@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4a6.ccs.ornl.gov kernel: [1914934.408156] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880322c5ec00 x1471904512502776/t0(0) o38->atlas2-MDT0000-lwp-OST028d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3i4.ccs.ornl.gov kernel: [1914938.271549] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:49:37 atlas-oss3e5.ccs.ornl.gov kernel: [1914933.590380] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8806cfdff400 x1471904510654808/t0(0) o38->atlas2-MDT0000-lwp-OST0024@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3e5.ccs.ornl.gov kernel: [1914933.682743] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss3g4.ccs.ornl.gov kernel: [1913542.845367] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880e99ed4400 x1471904510791364/t0(0) o38->atlas2-MDT0000-lwp-OST00c3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4g1.ccs.ornl.gov kernel: [1914929.434452] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880409c78400 x1471904508970400/t0(0) o38->atlas2-MDT0000-lwp-OST03d8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4d6.ccs.ornl.gov kernel: [1914932.439313] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88086fd32c00 x1471904519610420/t0(0) o38->atlas2-MDT0000-lwp-OST02a5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4d6.ccs.ornl.gov kernel: [1914932.531142] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss3c5.ccs.ornl.gov kernel: [1914938.961898] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880945065000 x1471904512466128/t0(0) o38->atlas2-MDT0000-lwp-OST0254@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3c2.ccs.ornl.gov kernel: [1914934.404798] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88063ed68000 x1471904511647940/t0(0) o38->atlas2-MDT0000-lwp-OST0251@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3f3.ccs.ornl.gov kernel: [1914945.886318] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880dda3b9000 x1471904510688092/t0(0) o38->atlas2-MDT0000-lwp-OST026a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3c6.ccs.ornl.gov kernel: [1914924.628114] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880a725cac00 x1471904508186880/t0(0) o38->atlas2-MDT0000-lwp-OST0255@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3c7.ccs.ornl.gov kernel: [1914933.244736] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8804587b1c00 x1471904510248676/t0(0) o38->atlas2-MDT0000-lwp-OST02e6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3e4.ccs.ornl.gov kernel: [1914946.619277] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8803697ff800 x1471904508538028/t0(0) o38->atlas2-MDT0000-lwp-OST0143@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3d3.ccs.ornl.gov kernel: [1914939.205993] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8804dece2400 x1471904510125724/t0(0) o38->atlas2-MDT0000-lwp-OST00aa@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3c5.ccs.ornl.gov kernel: [1914939.053355] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss4a3.ccs.ornl.gov kernel: [1914948.341113] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8806598d6400 x1471904514688896/t0(0) o38->atlas2-MDT0000-lwp-OST028a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4c6.ccs.ornl.gov kernel: [1914933.299809] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880e16901400 x1471904517937228/t0(0) o38->atlas2-MDT0000-lwp-OST029d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3c8.ccs.ornl.gov kernel: [1914937.435990] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88090c451400 x1471904507928048/t0(0) o38->atlas2-MDT0000-lwp-OST0017@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3e2.ccs.ornl.gov kernel: [1914931.785603] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880641554000 x1471904513179196/t0(0) o38->atlas2-MDT0000-lwp-OST0141@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3e2.ccs.ornl.gov kernel: [1914931.873916] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss4a7.ccs.ornl.gov kernel: [1914944.048119] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880836d9bc00 x1471904510936024/t0(0) o38->atlas2-MDT0000-lwp-OST004e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4a7.ccs.ornl.gov kernel: [1914944.139340] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss4a1.ccs.ornl.gov kernel: [1914930.344340] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880fc3b4d800 x1471904510345808/t0(0) o38->atlas2-MDT0000-lwp-OST0288@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4e6.ccs.ornl.gov kernel: [1914948.146695] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880815eca400 x1471904508408124/t0(0) o38->atlas2-MDT0000-lwp-OST021d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3h6.ccs.ornl.gov kernel: [1914932.780979] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880aabb46400 x1471904512216528/t0(0) o38->atlas2-MDT0000-lwp-OST01ed@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3d2.ccs.ornl.gov kernel: [1914934.126959] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880faf4a0c00 x1471904511214648/t0(0) o38->atlas2-MDT0000-lwp-OST00a9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3d2.ccs.ornl.gov kernel: [1914934.218122] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:49:37 atlas-oss3i6.ccs.ornl.gov kernel: [1914931.276202] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8804017df800 x1471904509634008/t0(0) o38->atlas2-MDT0000-lwp-OST0165@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3c4.ccs.ornl.gov kernel: [1914927.711694] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88077159d800 x1471904511832924/t0(0) o38->atlas2-MDT0000-lwp-OST01c3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3c4.ccs.ornl.gov kernel: [1914927.801418] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss3g3.ccs.ornl.gov kernel: [1913529.809668] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8802a56af800 x1471904516976764/t0(0) o38->atlas2-MDT0000-lwp-OST0392@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4d1.ccs.ornl.gov kernel: [1914928.456491] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880cd097ec00 x1471904516811712/t0(0) o38->atlas2-MDT0000-lwp-OST03c0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3i3.ccs.ornl.gov kernel: [1914927.904876] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880e86559800 x1471904518169268/t0(0) o38->atlas2-MDT0000-lwp-OST0162@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3a7.ccs.ornl.gov kernel: [1914930.561635] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8805f88eb400 x1471904518655632/t0(0) o38->atlas2-MDT0000-lwp-OST0096@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3g6.ccs.ornl.gov kernel: [1913540.808330] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88101ef93400 x1471904523473832/t0(0) o38->atlas2-MDT0000-lwp-OST0035@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3g6.ccs.ornl.gov kernel: [1913540.897919] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss4h3.ccs.ornl.gov kernel: [1914945.129037] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880ef1da4800 x1471904509650848/t0(0) o38->atlas2-MDT0000-lwp-OST0352@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3c3.ccs.ornl.gov kernel: [1914925.138016] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88072ad60800 x1471904511051444/t0(0) o38->atlas2-MDT0000-lwp-OST01c2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3d7.ccs.ornl.gov kernel: [1914933.316839] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880a01162800 x1471904512149452/t0(0) o38->atlas2-MDT0000-lwp-OST025e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4i1.ccs.ornl.gov kernel: [1914932.567977] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88087085f800 x1471904513319732/t0(0) o38->atlas2-MDT0000-lwp-OST01a8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4b4.ccs.ornl.gov kernel: [1914930.562913] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880860c05800 x1471904528872020/t0(0) o38->atlas2-MDT0000-lwp-OST00e3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4a8.ccs.ornl.gov kernel: [1914938.494632] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880e0968d000 x1471904507750768/t0(0) o38->atlas2-MDT0000-lwp-OST00df@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4a8.ccs.ornl.gov kernel: [1914938.587099] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss4f8.ccs.ornl.gov kernel: [1914931.707491] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8801a38f7800 x1471904511806032/t0(0) o38->atlas2-MDT0000-lwp-OST0077@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3f1.ccs.ornl.gov kernel: [1914936.289082] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88086b476000 x1471904508988432/t0(0) o38->atlas2-MDT0000-lwp-OST0268@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3b7.ccs.ornl.gov kernel: [1914937.347151] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880163c62800 x1471904519392644/t0(0) o38->atlas2-MDT0000-lwp-OST036e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3b7.ccs.ornl.gov kernel: [1914937.440410] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss3b1.ccs.ornl.gov kernel: [1914939.117055] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8807927a0800 x1471904516110752/t0(0) o38->atlas2-MDT0000-lwp-OST0098@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3b1.ccs.ornl.gov kernel: [1914939.209235] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss3i1.ccs.ornl.gov kernel: [1914932.622610] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8806c41e5400 x1471904519755408/t0(0) o38->atlas2-MDT0000-lwp-OST01f0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4d8.ccs.ornl.gov kernel: [1914945.533575] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8804b0d97000 x1471904516222992/t0(0) o38->atlas2-MDT0000-lwp-OST02a7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3b8.ccs.ornl.gov kernel: [1914946.666865] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff880ff4028000 x1471904516406780/t0(0) o38->atlas2-MDT0000-lwp-OST009f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss4d8.ccs.ornl.gov kernel: [1914945.624896] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:37 atlas-oss3d5.ccs.ornl.gov kernel: [1914936.019711] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8802c9c7d400 x1471904509642028/t0(0) o38->atlas2-MDT0000-lwp-OST001c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3b4.ccs.ornl.gov kernel: [1914944.185510] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff8810166cf000 x1471904510548560/t0(0) o38->atlas2-MDT0000-lwp-OST02db@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:37 atlas-oss3i2.ccs.ornl.gov kernel: [1914938.730951] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629897/real 1405629897] req@ffff88047c07ec00 x1471904514016264/t0(0) o38->atlas2-MDT0000-lwp-OST03a1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3i4.ccs.ornl.gov kernel: [1914956.827975] Lustre: 27273:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8806ab2a6000 x1471904510470324/t0(0) o400->atlas2-MDT0000-lwp-OST03a3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4i7.ccs.ornl.gov kernel: [1914964.250907] Lustre: 25151:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880915998000 x1471904511375036/t0(0) o400->atlas2-MDT0000-lwp-OST008e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4i7.ccs.ornl.gov kernel: [1914964.341185] Lustre: 25151:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4i4.ccs.ornl.gov kernel: [1914955.877055] Lustre: 22171:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880cedcd7400 x1471904516513372/t0(0) o400->atlas2-MDT0000-lwp-OST02cb@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4i4.ccs.ornl.gov kernel: [1914955.969939] Lustre: 22171:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4f6.ccs.ornl.gov kernel: [1914945.766232] Lustre: 22288:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880c1ed2f000 x1471904511708484/t0(0) o400->atlas2-MDT0000-lwp-OST0105@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4f6.ccs.ornl.gov kernel: [1914945.855635] Lustre: 22288:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:49:56 atlas-oss3e3.ccs.ornl.gov kernel: [1914947.678476] Lustre: 23063:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8807324c5800 x1471904510468384/t0(0) o400->atlas2-MDT0000-lwp-OST0142@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4b7.ccs.ornl.gov kernel: [1914945.058573] Lustre: 23156:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880e0f7d9400 x1471904509377236/t0(0) o400->atlas2-MDT0000-lwp-OST03b6@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4b7.ccs.ornl.gov kernel: [1914945.149070] Lustre: 23156:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4i3.ccs.ornl.gov kernel: [1914943.236364] Lustre: 25965:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880735ba4c00 x1471904523159516/t0(0) o400->atlas2-MDT0000-lwp-OST01aa@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4g6.ccs.ornl.gov kernel: [1914960.683096] Lustre: 24690:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88046724f800 x1471904507898620/t0(0) o400->atlas2-MDT0000-lwp-OST019d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4d7.ccs.ornl.gov kernel: [1914951.704316] Lustre: 22282:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880b32eba400 x1471904514719912/t0(0) o400->atlas2-MDT0000-lwp-OST00f6@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3h5.ccs.ornl.gov kernel: [1914970.710076] Lustre: 24602:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88011e9d5400 x1471904507806512/t0(0) o400->atlas2-MDT0000-lwp-OST003c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3h5.ccs.ornl.gov kernel: [1914970.802116] Lustre: 24602:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:56 atlas-oss3f6.ccs.ornl.gov kernel: [801655.597675] Lustre: 79675:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880e9ff87800 x1473076856558508/t0(0) o400->atlas2-MDT0000-lwp-OST002d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4b6.ccs.ornl.gov kernel: [1914948.637548] Lustre: 26095:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88021dd38800 x1471904511213512/t0(0) o400->atlas2-MDT0000-lwp-OST03b5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3e7.ccs.ornl.gov kernel: [1914953.416635] Lustre: 25879:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880efe89b000 x1471904508195148/t0(0) o400->atlas2-MDT0000-lwp-OST0266@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3e7.ccs.ornl.gov kernel: [1914953.507833] Lustre: 25879:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:49:56 atlas-oss3c1.ccs.ornl.gov kernel: [1914953.152637] Lustre: 23677:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88016be41400 x1471904511392900/t0(0) o400->atlas2-MDT0000-lwp-OST00a0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3c1.ccs.ornl.gov kernel: [1914953.243846] Lustre: 23677:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4f1.ccs.ornl.gov kernel: [1914955.204925] Lustre: 25959:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8802475cb400 x1471904513370984/t0(0) o400->atlas2-MDT0000-lwp-OST0070@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4f1.ccs.ornl.gov kernel: [1914955.296495] Lustre: 25959:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4f4.ccs.ornl.gov kernel: [1914951.496556] Lustre: 22162:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8801c2c7dc00 x1471904516980196/t0(0) o400->atlas2-MDT0000-lwp-OST02b3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4f4.ccs.ornl.gov kernel: [1914951.586589] Lustre: 22162:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:49:56 atlas-oss4d3.ccs.ornl.gov kernel: [1914947.505008] Lustre: 22338:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880f32218000 x1471904516978196/t0(0) o400->atlas2-MDT0000-lwp-OST0182@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4h4.ccs.ornl.gov kernel: [1914970.442883] Lustre: 25831:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880418398800 x1471904512334300/t0(0) o400->atlas2-MDT0000-lwp-OST02c3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4g6.ccs.ornl.gov kernel: [1914960.767501] Lustre: 24690:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:49:56 atlas-oss4c5.ccs.ornl.gov kernel: [1914946.789539] Lustre: 26660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88067cdad800 x1471904510171160/t0(0) o400->atlas2-MDT0000-lwp-OST020c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4c5.ccs.ornl.gov kernel: [1914946.882499] Lustre: 26660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:49:56 atlas-oss4c3.ccs.ornl.gov kernel: [1914947.828362] Lustre: 24967:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880fd061bc00 x1471904512246000/t0(0) o400->atlas2-MDT0000-lwp-OST005a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4f2.ccs.ornl.gov kernel: [1914954.620894] Lustre: 25728:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880516c4b800 x1471904508910608/t0(0) o400->atlas2-MDT0000-lwp-OST0071@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4f2.ccs.ornl.gov kernel: [1914954.706400] Lustre: 25728:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss3g8.ccs.ornl.gov kernel: [1913549.102006] Lustre: 13443:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88092957cc00 x1471904513602340/t0(0) o400->atlas2-MDT0000-lwp-OST0037@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3h8.ccs.ornl.gov kernel: [1914947.923351] Lustre: 22181:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88074a0bfc00 x1471904514772756/t0(0) o400->atlas2-MDT0000-lwp-OST01ef@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3h8.ccs.ornl.gov kernel: [1914948.015485] Lustre: 22181:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss3d6.ccs.ornl.gov kernel: [1914965.439222] Lustre: 25611:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8808cb007c00 x1471904511183324/t0(0) o400->atlas2-MDT0000-lwp-OST02ed@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3d6.ccs.ornl.gov kernel: [1914965.532194] Lustre: 25611:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:49:56 atlas-oss4i8.ccs.ornl.gov kernel: [1914964.754062] Lustre: 26097:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8808e5afb400 x1471904520247056/t0(0) o400->atlas2-MDT0000-lwp-OST01af@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4i8.ccs.ornl.gov kernel: [1914964.846750] Lustre: 26097:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4d3.ccs.ornl.gov kernel: [1914947.597369] Lustre: 22338:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4a5.ccs.ornl.gov kernel: [1914970.020923] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880bf1f63c00 x1471904519454080/t0(0) o400->atlas2-MDT0000-lwp-OST01fc@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4a5.ccs.ornl.gov kernel: [1914970.113698] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4h4.ccs.ornl.gov kernel: [1914970.533120] Lustre: 25831:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:49:56 atlas-oss3e1.ccs.ornl.gov kernel: [1914953.823990] Lustre: 23612:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88014bb39800 x1471904509694268/t0(0) o400->atlas2-MDT0000-lwp-OST0020@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4c8.ccs.ornl.gov kernel: [1914942.942977] Lustre: 22055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88054772f000 x1471904512196860/t0(0) o400->atlas2-MDT0000-lwp-OST032f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4c8.ccs.ornl.gov kernel: [1914943.034103] Lustre: 22055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss3f5.ccs.ornl.gov kernel: [801642.984411] Lustre: 79205:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880ccc323400 x1473076856406228/t0(0) o400->atlas2-MDT0000-lwp-OST01dc@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4e7.ccs.ornl.gov kernel: [1914948.492862] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88044aa86000 x1471904522621448/t0(0) o400->atlas2-MDT0000-lwp-OST021e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4e7.ccs.ornl.gov kernel: [1914948.584518] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss3g8.ccs.ornl.gov kernel: [1913549.192910] Lustre: 13443:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:56 atlas-oss4i6.ccs.ornl.gov kernel: [1914964.465066] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88070ec77c00 x1471904510597520/t0(0) o400->atlas2-MDT0000-lwp-OST03ed@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3f8.ccs.ornl.gov kernel: [801653.485947] Lustre: 81099:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880572ff9000 x1473076861320868/t0(0) o400->atlas2-MDT0000-lwp-OST00bf@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4i6.ccs.ornl.gov kernel: [1914964.557871] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss3f8.ccs.ornl.gov kernel: [801653.578241] Lustre: 81099:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:56 atlas-oss4c2.ccs.ornl.gov kernel: [1914955.030650] Lustre: 25987:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880144280000 x1471904508570588/t0(0) o400->atlas2-MDT0000-lwp-OST0059@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3i7.ccs.ornl.gov kernel: [1914948.265513] Lustre: 24586:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff881026e62400 x1471904508389224/t0(0) o400->atlas2-MDT0000-lwp-OST00d6@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3i7.ccs.ornl.gov kernel: [1914948.355853] Lustre: 24586:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss3e1.ccs.ornl.gov kernel: [1914953.915575] Lustre: 23612:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4e1.ccs.ornl.gov kernel: [1914948.988606] Lustre: 26047:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880d25b8d400 x1471904515144676/t0(0) o400->atlas2-MDT0000-lwp-OST0068@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4e1.ccs.ornl.gov kernel: [1914949.077588] Lustre: 26047:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4g8.ccs.ornl.gov kernel: [1914950.222959] Lustre: 25556:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88054c288000 x1471904521746748/t0(0) o400->atlas2-MDT0000-lwp-OST007f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3h3.ccs.ornl.gov kernel: [1914946.729323] Lustre: 26635:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8806efd42400 x1471904512413528/t0(0) o400->atlas2-MDT0000-lwp-OST003a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4d4.ccs.ornl.gov kernel: [1914971.467284] Lustre: 27389:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8801a900f400 x1471904513679100/t0(0) o400->atlas2-MDT0000-lwp-OST03c3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3h3.ccs.ornl.gov kernel: [1914946.822276] Lustre: 26635:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss3a1.ccs.ornl.gov kernel: [1914945.961403] Lustre: 22442:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880d392a2000 x1471904525931688/t0(0) o400->atlas2-MDT0000-lwp-OST0090@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3f7.ccs.ornl.gov kernel: [801282.199507] Lustre: 78676:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880731b5e800 x1473076863306112/t0(0) o400->atlas2-MDT0000-lwp-OST002e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3f7.ccs.ornl.gov kernel: [801282.290362] Lustre: 78676:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:49:56 atlas-oss3h7.ccs.ornl.gov kernel: [1914957.850536] Lustre: 25086:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88081ba93c00 x1471904510332872/t0(0) o400->atlas2-MDT0000-lwp-OST015e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3h7.ccs.ornl.gov kernel: [1914957.943196] Lustre: 25086:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4e2.ccs.ornl.gov kernel: [1914964.940477] Lustre: 22173:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880f10c9a000 x1471904514146264/t0(0) o400->atlas2-MDT0000-lwp-OST0069@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4e2.ccs.ornl.gov kernel: [1914965.032586] Lustre: 22173:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:49:56 atlas-oss4h8.ccs.ornl.gov kernel: [1914942.084729] Lustre: 24627:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880015817c00 x1471904514267124/t0(0) o400->atlas2-MDT0000-lwp-OST0087@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4c1.ccs.ornl.gov kernel: [1914956.642077] Lustre: 22143:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8801da92a800 x1471904516097428/t0(0) o400->atlas2-MDT0000-lwp-OST0058@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3b3.ccs.ornl.gov kernel: [1914944.131423] Lustre: 25590:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8804593dc800 x1471904513820352/t0(0) o400->atlas2-MDT0000-lwp-OST009a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3b3.ccs.ornl.gov kernel: [1914944.221543] Lustre: 25590:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4b1.ccs.ornl.gov kernel: [1914955.650098] Lustre: 25872:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880239c70000 x1471904507949016/t0(0) o400->atlas2-MDT0000-lwp-OST00e0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4b1.ccs.ornl.gov kernel: [1914955.742052] Lustre: 25872:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4d4.ccs.ornl.gov kernel: [1914971.559368] Lustre: 27389:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4g8.ccs.ornl.gov kernel: [1914950.314593] Lustre: 25556:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4a2.ccs.ornl.gov kernel: [1914965.797974] Lustre: 25782:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880626324000 x1471904510895652/t0(0) o400->atlas2-MDT0000-lwp-OST0319@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4a2.ccs.ornl.gov kernel: [1914965.888339] Lustre: 25782:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4h8.ccs.ornl.gov kernel: [1914942.176151] Lustre: 24627:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:49:56 atlas-oss4c1.ccs.ornl.gov kernel: [1914956.734963] Lustre: 22143:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:49:56 atlas-oss4g3.ccs.ornl.gov kernel: [1914962.329383] Lustre: 25832:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88103de51c00 x1471904517610152/t0(0) o400->atlas2-MDT0000-lwp-OST007a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3a5.ccs.ornl.gov kernel: [1914944.723256] Lustre: 22234:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8803b868c000 x1471904511738388/t0(0) o400->atlas2-MDT0000-lwp-OST0004@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3a6.ccs.ornl.gov kernel: [1914953.824311] Lustre: 25782:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8802b96cf000 x1471904512394564/t0(0) o400->atlas2-MDT0000-lwp-OST0005@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3a6.ccs.ornl.gov kernel: [1914953.915871] Lustre: 25782:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss3a5.ccs.ornl.gov kernel: [1914944.815432] Lustre: 22234:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:49:56 atlas-oss3b5.ccs.ornl.gov kernel: [1914964.233538] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880026decc00 x1471904514716904/t0(0) o400->atlas2-MDT0000-lwp-OST024c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3b5.ccs.ornl.gov kernel: [1914964.335027] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4g7.ccs.ornl.gov kernel: [1914973.534854] Lustre: 26066:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880b8b1b3800 x1471904508873504/t0(0) o400->atlas2-MDT0000-lwp-OST010e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4g7.ccs.ornl.gov kernel: [1914973.625944] Lustre: 26066:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss3h2.ccs.ornl.gov kernel: [1914954.285436] Lustre: 20919:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880ffff45800 x1471904518760428/t0(0) o400->atlas2-MDT0000-lwp-OST0039@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3h2.ccs.ornl.gov kernel: [1914954.377305] Lustre: 20919:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4b8.ccs.ornl.gov kernel: [1914953.108657] Lustre: 25794:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88103376e400 x1471904512563848/t0(0) o400->atlas2-MDT0000-lwp-OST0207@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4b8.ccs.ornl.gov kernel: [1914953.200739] Lustre: 25794:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:49:56 atlas-oss3h4.ccs.ornl.gov kernel: [1914955.056236] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff881071f82400 x1471904509245184/t0(0) o400->atlas2-MDT0000-lwp-OST003b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3h4.ccs.ornl.gov kernel: [1914955.146572] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss3h1.ccs.ornl.gov kernel: [1914939.992716] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880142cd3000 x1471904517765820/t0(0) o400->atlas2-MDT0000-lwp-OST0038@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3h1.ccs.ornl.gov kernel: [1914940.085320] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss4e8.ccs.ornl.gov kernel: [1914964.237687] Lustre: 24605:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880457992000 x1471904518472716/t0(0) o400->atlas2-MDT0000-lwp-OST006f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4e8.ccs.ornl.gov kernel: [1914964.328644] Lustre: 24605:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:49:56 atlas-oss4f5.ccs.ornl.gov kernel: [1914949.117718] Lustre: 22152:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880987a82800 x1471904509390856/t0(0) o400->atlas2-MDT0000-lwp-OST0104@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4f5.ccs.ornl.gov kernel: [1914949.117724] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880bdc57e800 x1471904509390852/t0(0) o400->atlas2-MDT0000-lwp-OST0074@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4f5.ccs.ornl.gov kernel: [1914949.117728] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:56 atlas-oss3e6.ccs.ornl.gov kernel: [1914949.027112] Lustre: 22106:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8809610ddc00 x1471904520534556/t0(0) o400->atlas2-MDT0000-lwp-OST0025@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss3e6.ccs.ornl.gov kernel: [1914949.119661] Lustre: 22106:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:49:56 atlas-oss4g5.ccs.ornl.gov kernel: [1914951.763745] Lustre: 26058:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880841722c00 x1471904510456680/t0(0) o400->atlas2-MDT0000-lwp-OST022c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4g5.ccs.ornl.gov kernel: [1914951.848340] Lustre: 26058:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:49:56 atlas-oss4e4.ccs.ornl.gov kernel: [1914960.357757] Lustre: 22204:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8802ef9e2000 x1471904515374252/t0(0) o400->atlas2-MDT0000-lwp-OST006b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4e4.ccs.ornl.gov kernel: [1914960.442547] Lustre: 22204:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:49:56 atlas-oss4g2.ccs.ornl.gov kernel: [1914962.425716] Lustre: 22131:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8806eda6e000 x1471904509228920/t0(0) o400->atlas2-MDT0000-lwp-OST0349@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4g2.ccs.ornl.gov kernel: [1914962.515912] Lustre: 22131:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:56 atlas-oss4i2.ccs.ornl.gov kernel: [1914974.529416] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880055e4f000 x1471904514634476/t0(0) o400->atlas2-MDT0000-lwp-OST0359@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4i2.ccs.ornl.gov kernel: [1914974.617651] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:56 atlas-oss4d2.ccs.ornl.gov kernel: [1914956.775212] Lustre: 27441:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88068298b800 x1471904512220912/t0(0) o400->atlas2-MDT0000-lwp-OST03c1@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4d2.ccs.ornl.gov kernel: [1914956.865858] Lustre: 27441:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:56 atlas-oss4h6.ccs.ornl.gov kernel: [1914950.479294] Lustre: 25965:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880a85ea5000 x1471904515532052/t0(0) o400->atlas2-MDT0000-lwp-OST03e5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4a4.ccs.ornl.gov kernel: [1914953.232814] Lustre: 27361:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8807620d9c00 x1471904509591020/t0(0) o400->atlas2-MDT0000-lwp-OST004b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4a4.ccs.ornl.gov kernel: [1914953.322110] Lustre: 27361:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:49:56 atlas-oss4b3.ccs.ornl.gov kernel: [1914945.994877] Lustre: 22296:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8803a57a9c00 x1471904509180452/t0(0) o400->atlas2-MDT0000-lwp-OST00e2@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4h7.ccs.ornl.gov kernel: [1914959.958876] Lustre: 26108:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88031c642c00 x1471904512615088/t0(0) o400->atlas2-MDT0000-lwp-OST0236@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4h7.ccs.ornl.gov kernel: [1914960.056920] Lustre: 26108:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:56 atlas-oss4f7.ccs.ornl.gov kernel: [1914950.756557] Lustre: 25988:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff8802bff0bc00 x1471904510454980/t0(0) o400->atlas2-MDT0000-lwp-OST0196@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4f7.ccs.ornl.gov kernel: [1914950.849515] Lustre: 25988:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:56 atlas-oss4h2.ccs.ornl.gov kernel: [1914972.920896] Lustre: 26011:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff880646ce6400 x1471904508411068/t0(0) o400->atlas2-MDT0000-lwp-OST0111@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:56 atlas-oss4b3.ccs.ornl.gov kernel: [1914946.080573] Lustre: 22296:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:56 atlas-oss4h6.ccs.ornl.gov kernel: [1914950.568154] Lustre: 25965:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:56 atlas-oss4h2.ccs.ornl.gov kernel: [1914973.011537] Lustre: 26011:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:56 atlas-oss3g5.ccs.ornl.gov kernel: [1913553.424289] Lustre: 13572:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629629/real 1405629629] req@ffff88063c600c00 x1471904518249032/t0(0) o400->atlas2-MDT0000-lwp-OST0394@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630196 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3g5.ccs.ornl.gov kernel: [1913553.515254] Lustre: 13572:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:57 atlas-oss4g4.ccs.ornl.gov kernel: [1914966.408732] Lustre: 25821:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8807f7d0bc00 x1471904515530612/t0(0) o400->atlas2-MDT0000-lwp-OST034b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4g4.ccs.ornl.gov kernel: [1914966.493553] Lustre: 25821:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:49:57 atlas-oss4b5.ccs.ornl.gov kernel: [1914969.534106] Lustre: 25794:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880415272800 x1471904518365596/t0(0) o400->atlas2-MDT0000-lwp-OST0324@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4a6.ccs.ornl.gov kernel: [1914954.042399] Lustre: 25597:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff88043cdc5c00 x1471904512502712/t0(0) o400->atlas2-MDT0000-lwp-OST031d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4a6.ccs.ornl.gov kernel: [1914954.135659] Lustre: 25597:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:49:57 atlas-oss3c7.ccs.ornl.gov kernel: [1914952.866000] Lustre: 25751:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff88063ecc4c00 x1471904510248620/t0(0) o400->atlas2-MDT0000-lwp-OST0376@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3c7.ccs.ornl.gov kernel: [1914952.953352] Lustre: 25751:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:49:57 atlas-oss3d3.ccs.ornl.gov kernel: [1914958.814303] Lustre: 25543:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880460ea3800 x1471904510125648/t0(0) o400->atlas2-MDT0000-lwp-OST013a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3d3.ccs.ornl.gov kernel: [1914958.905153] Lustre: 25543:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:49:57 atlas-oss4c6.ccs.ornl.gov kernel: [1914952.909041] Lustre: 22197:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880d3e0b8c00 x1471904517937152/t0(0) o400->atlas2-MDT0000-lwp-OST017d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4c6.ccs.ornl.gov kernel: [1914952.999620] Lustre: 22197:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss3g2.ccs.ornl.gov kernel: [1913548.631843] Lustre: 13995:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880870731800 x1471904509036064/t0(0) o400->atlas2-MDT0000-lwp-OST0301@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3g2.ccs.ornl.gov kernel: [1913548.723488] Lustre: 13995:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:49:57 atlas-oss4f3.ccs.ornl.gov kernel: [1914946.726889] Lustre: 26006:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8803cdb6fc00 x1471904513977088/t0(0) o400->atlas2-MDT0000-lwp-OST0342@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4f3.ccs.ornl.gov kernel: [1914946.817498] Lustre: 26006:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss4d6.ccs.ornl.gov kernel: [1914952.093549] Lustre: 25548:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880870d25400 x1471904519610340/t0(0) o400->atlas2-MDT0000-lwp-OST00f5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4d6.ccs.ornl.gov kernel: [1914952.178400] Lustre: 25548:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss4b5.ccs.ornl.gov kernel: [1914969.618834] Lustre: 25794:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss3e5.ccs.ornl.gov kernel: [1914953.253618] Lustre: 25948:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8807480b2c00 x1471904510654728/t0(0) o400->atlas2-MDT0000-lwp-OST00b4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3e5.ccs.ornl.gov kernel: [1914953.340101] Lustre: 25948:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss3e4.ccs.ornl.gov kernel: [1914966.246675] Lustre: 22110:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880408129400 x1471904508537960/t0(0) o400->atlas2-MDT0000-lwp-OST0263@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4g1.ccs.ornl.gov kernel: [1914949.094677] Lustre: 25811:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880698370800 x1471904508970320/t0(0) o400->atlas2-MDT0000-lwp-OST0108@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4g1.ccs.ornl.gov kernel: [1914949.182513] Lustre: 25811:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss4a1.ccs.ornl.gov kernel: [1914949.968547] Lustre: 25990:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff88100942ec00 x1471904510345728/t0(0) o400->atlas2-MDT0000-lwp-OST00d8@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4a1.ccs.ornl.gov kernel: [1914950.055379] Lustre: 25990:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss3a2.ccs.ornl.gov kernel: [1914951.555119] Lustre: 25624:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880b590d1000 x1471904510524224/t0(0) o400->atlas2-MDT0000-lwp-OST0361@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3a2.ccs.ornl.gov kernel: [1914951.644232] Lustre: 25624:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss4e3.ccs.ornl.gov kernel: [1914957.831140] Lustre: 27363:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8805e3412400 x1471904515523664/t0(0) o400->atlas2-MDT0000-lwp-OST03ca@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4e3.ccs.ornl.gov kernel: [1914957.923982] Lustre: 27363:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss3g1.ccs.ornl.gov kernel: [1913541.986273] Lustre: 13839:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880e5392e400 x1471904510299568/t0(0) o400->atlas2-MDT0000-lwp-OST01e0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3g1.ccs.ornl.gov kernel: [1913542.079029] Lustre: 13839:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss4e6.ccs.ornl.gov kernel: [1914967.778108] Lustre: 25707:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff88013d878c00 x1471904508408060/t0(0) o400->atlas2-MDT0000-lwp-OST02ad@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4e6.ccs.ornl.gov kernel: [1914967.866372] Lustre: 25707:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:49:57 atlas-oss4a7.ccs.ornl.gov kernel: [1914963.672458] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8805427f5000 x1471904510935940/t0(0) o400->atlas2-MDT0000-lwp-OST004e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4a7.ccs.ornl.gov kernel: [1914963.776130] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss4d5.ccs.ornl.gov kernel: [1914964.539446] Lustre: 26094:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880f7eab8000 x1471904519136760/t0(0) o400->atlas2-MDT0000-lwp-OST02a4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4d5.ccs.ornl.gov kernel: [1914964.629227] Lustre: 26094:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:49:57 atlas-oss4a3.ccs.ornl.gov kernel: [1914967.995552] Lustre: 28207:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff88058db85800 x1471904514688832/t0(0) o400->atlas2-MDT0000-lwp-OST031a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4a3.ccs.ornl.gov kernel: [1914968.082837] Lustre: 28207:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:49:57 atlas-oss3e4.ccs.ornl.gov kernel: [1914966.337942] Lustre: 22110:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss3c8.ccs.ornl.gov kernel: [1914957.091259] Lustre: 26031:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880cb7312800 x1471904507927972/t0(0) o400->atlas2-MDT0000-lwp-OST0137@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3c8.ccs.ornl.gov kernel: [1914957.175754] Lustre: 26031:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss3d8.ccs.ornl.gov kernel: [1914958.592888] Lustre: 24587:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880f44428c00 x1471904508710616/t0(0) o400->atlas2-MDT0000-lwp-OST037f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3d8.ccs.ornl.gov kernel: [1914958.683596] Lustre: 24587:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss4i5.ccs.ornl.gov kernel: [1914959.994192] Lustre: 25966:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8807519dd400 x1471904512720336/t0(0) o400->atlas2-MDT0000-lwp-OST01ac@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4i5.ccs.ornl.gov kernel: [1914960.083139] Lustre: 25966:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss4d1.ccs.ornl.gov kernel: [1914948.080576] Lustre: 22257:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880eea387c00 x1471904516811636/t0(0) o400->atlas2-MDT0000-lwp-OST0180@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4d1.ccs.ornl.gov kernel: [1914948.166114] Lustre: 22257:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:49:57 atlas-oss3e2.ccs.ornl.gov kernel: [1914951.445845] Lustre: 27203:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8805b8511400 x1471904513179124/t0(0) o400->atlas2-MDT0000-lwp-OST01d1@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3e2.ccs.ornl.gov kernel: [1914951.530752] Lustre: 27203:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:49:57 atlas-oss4b4.ccs.ornl.gov kernel: [1914950.167142] Lustre: 26021:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880435e10000 x1471904528871940/t0(0) o400->atlas2-MDT0000-lwp-OST00e3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4b4.ccs.ornl.gov kernel: [1914950.252596] Lustre: 26021:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss3i8.ccs.ornl.gov kernel: [1914965.944647] Lustre: 22179:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8803aad0cc00 x1471904515355392/t0(0) o400->atlas2-MDT0000-lwp-OST0047@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3i8.ccs.ornl.gov kernel: [1914966.031487] Lustre: 22179:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss4h5.ccs.ornl.gov kernel: [1914954.555174] Lustre: 25672:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8805c3e30000 x1471904509313408/t0(0) o400->atlas2-MDT0000-lwp-OST03e4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4h5.ccs.ornl.gov kernel: [1914954.643877] Lustre: 25672:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 16:49:57 atlas-oss3i3.ccs.ornl.gov kernel: [1914947.532106] Lustre: 22131:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff881016a7dc00 x1471904518169204/t0(0) o400->atlas2-MDT0000-lwp-OST0312@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3i3.ccs.ornl.gov kernel: [1914947.623677] Lustre: 22131:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:49:57 atlas-oss3a4.ccs.ornl.gov kernel: [1914942.053641] Lustre: 22194:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880486c1bc00 x1471904512538716/t0(0) o400->atlas2-MDT0000-lwp-OST01b3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3a4.ccs.ornl.gov kernel: [1914942.140010] Lustre: 22194:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss3d2.ccs.ornl.gov kernel: [1914953.774232] Lustre: 25974:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8807b8034400 x1471904511214592/t0(0) o400->atlas2-MDT0000-lwp-OST0379@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3d2.ccs.ornl.gov kernel: [1914953.865879] Lustre: 25974:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:49:57 atlas-oss3d1.ccs.ornl.gov kernel: [1914944.738635] Lustre: 25664:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880218b8d400 x1471904515778236/t0(0) o400->atlas2-MDT0000-lwp-OST00a8@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3d1.ccs.ornl.gov kernel: [1914944.825766] Lustre: 25664:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:49:57 atlas-oss4d8.ccs.ornl.gov kernel: [1914965.112941] Lustre: 22119:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8802d9b6dc00 x1471904516222920/t0(0) o400->atlas2-MDT0000-lwp-OST0187@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3h6.ccs.ornl.gov kernel: [1914952.436261] Lustre: 25748:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880e0f65f800 x1471904512216456/t0(0) o400->atlas2-MDT0000-lwp-OST01ed@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4d8.ccs.ornl.gov kernel: [1914965.202081] Lustre: 22119:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss3h6.ccs.ornl.gov kernel: [1914952.530931] Lustre: 25748:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss3f2.ccs.ornl.gov kernel: [1914947.202484] Lustre: 25712:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880a43dde400 x1471904513672460/t0(0) o400->atlas2-MDT0000-lwp-OST0269@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3f2.ccs.ornl.gov kernel: [1914947.292902] Lustre: 25712:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss3a3.ccs.ornl.gov kernel: [1914949.310523] Lustre: 27239:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880c8fa23000 x1471904526379848/t0(0) o400->atlas2-MDT0000-lwp-OST01b2@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3a3.ccs.ornl.gov kernel: [1914949.395269] Lustre: 27239:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss3g6.ccs.ornl.gov kernel: [1913560.438718] Lustre: 13995:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880488ded400 x1471904523473772/t0(0) o400->atlas2-MDT0000-lwp-OST0395@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3g6.ccs.ornl.gov kernel: [1913560.525925] Lustre: 13995:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss3e8.ccs.ornl.gov kernel: [1914952.188430] Lustre: 25795:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8806158d7c00 x1471904515139528/t0(0) o400->atlas2-MDT0000-lwp-OST0387@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3e8.ccs.ornl.gov kernel: [1914952.280831] Lustre: 25795:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:49:57 atlas-oss4a8.ccs.ornl.gov kernel: [1914958.114952] Lustre: 25956:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880bbfcef400 x1471904507750704/t0(0) o400->atlas2-MDT0000-lwp-OST031f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4h3.ccs.ornl.gov kernel: [1914964.756370] Lustre: 26065:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff88002708d800 x1471904509650788/t0(0) o400->atlas2-MDT0000-lwp-OST03e2@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4h3.ccs.ornl.gov kernel: [1914964.848018] Lustre: 26065:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss3i6.ccs.ornl.gov kernel: [1914950.934430] Lustre: 25707:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8807435ed000 x1471904509633944/t0(0) o400->atlas2-MDT0000-lwp-OST0315@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3i6.ccs.ornl.gov kernel: [1914951.025678] Lustre: 25707:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:49:57 atlas-oss3c3.ccs.ornl.gov kernel: [1914944.767180] Lustre: 22229:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8801eeb05800 x1471904511051376/t0(0) o400->atlas2-MDT0000-lwp-OST01c2@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3c3.ccs.ornl.gov kernel: [1914944.857360] Lustre: 22229:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss3d5.ccs.ornl.gov kernel: [1914955.561976] Lustre: 22191:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8804bc8ec400 x1471904509641952/t0(0) o400->atlas2-MDT0000-lwp-OST00ac@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3d5.ccs.ornl.gov kernel: [1914955.649377] Lustre: 22191:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss4h1.ccs.ornl.gov kernel: [1914945.958291] Lustre: 25864:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880da9d1f000 x1471904511603044/t0(0) o400->atlas2-MDT0000-lwp-OST0080@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4h1.ccs.ornl.gov kernel: [1914946.049172] Lustre: 25864:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss3c4.ccs.ornl.gov kernel: [1914947.370942] Lustre: 25948:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8803a9669000 x1471904511832864/t0(0) o400->atlas2-MDT0000-lwp-OST0373@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3c4.ccs.ornl.gov kernel: [1914947.458898] Lustre: 25948:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:49:57 atlas-oss3b7.ccs.ornl.gov kernel: [1914956.965422] Lustre: 26038:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8801ac057400 x1471904519392584/t0(0) o400->atlas2-MDT0000-lwp-OST036e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3a7.ccs.ornl.gov kernel: [1914950.213906] Lustre: 22081:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff88077f452c00 x1471904518655572/t0(0) o400->atlas2-MDT0000-lwp-OST0366@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3a7.ccs.ornl.gov kernel: [1914950.301084] Lustre: 22081:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:49:57 atlas-oss3g3.ccs.ornl.gov kernel: [1913549.470920] Lustre: 13963:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880c0cb0e800 x1471904516976692/t0(0) o400->atlas2-MDT0000-lwp-OST0272@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3g3.ccs.ornl.gov kernel: [1913549.558242] Lustre: 13963:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:49:57 atlas-oss3b1.ccs.ornl.gov kernel: [1914958.737392] Lustre: 22327:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff88081a922400 x1471904516110672/t0(0) o400->atlas2-MDT0000-lwp-OST0098@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3b1.ccs.ornl.gov kernel: [1914958.826889] Lustre: 22327:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss4i1.ccs.ornl.gov kernel: [1914952.204190] Lustre: 25774:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8808564a0800 x1471904513319648/t0(0) o400->atlas2-MDT0000-lwp-OST0088@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4i1.ccs.ornl.gov kernel: [1914952.296789] Lustre: 25774:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss3b6.ccs.ornl.gov kernel: [1914950.322014] Lustre: 25967:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880382e73800 x1471904512275316/t0(0) o400->atlas2-MDT0000-lwp-OST024d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3f3.ccs.ornl.gov kernel: [1914965.611688] Lustre: 26010:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8810240e0800 x1471904510688016/t0(0) o400->atlas2-MDT0000-lwp-OST014a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3b6.ccs.ornl.gov kernel: [1914950.410118] Lustre: 25967:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss3f3.ccs.ornl.gov kernel: [1914965.704746] Lustre: 26010:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss3g4.ccs.ornl.gov kernel: [1913562.589834] Lustre: 13612:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880b9a6a3c00 x1471904510791296/t0(0) o400->atlas2-MDT0000-lwp-OST01e3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3g4.ccs.ornl.gov kernel: [1913562.674724] Lustre: 13612:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss3i1.ccs.ornl.gov kernel: [1914952.246870] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880102ce3000 x1471904519755340/t0(0) o400->atlas2-MDT0000-lwp-OST0280@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3i1.ccs.ornl.gov kernel: [1914952.332152] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:49:57 atlas-oss3b2.ccs.ornl.gov kernel: [1914952.097335] Lustre: 25276:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880d5173d000 x1471904511916944/t0(0) o400->atlas2-MDT0000-lwp-OST02d9@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3b2.ccs.ornl.gov kernel: [1914952.183854] Lustre: 25276:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss4a8.ccs.ornl.gov kernel: [1914958.204613] Lustre: 25956:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss3c2.ccs.ornl.gov kernel: [1914954.159103] Lustre: 25867:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff8806c5a04800 x1471904511647880/t0(0) o400->atlas2-MDT0000-lwp-OST0371@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3c2.ccs.ornl.gov kernel: [1914954.243903] Lustre: 25867:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss3d7.ccs.ornl.gov kernel: [1914952.976165] Lustre: 22228:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880b84a6ec00 x1471904512149388/t0(0) o400->atlas2-MDT0000-lwp-OST02ee@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3d7.ccs.ornl.gov kernel: [1914953.066089] Lustre: 22228:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:49:57 atlas-oss3c5.ccs.ornl.gov kernel: [1914958.721292] Lustre: 25805:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880e1cebec00 x1471904512466060/t0(0) o400->atlas2-MDT0000-lwp-OST0254@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3c5.ccs.ornl.gov kernel: [1914958.811067] Lustre: 25805:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:49:57 atlas-oss4f8.ccs.ornl.gov kernel: [1914951.366701] Lustre: 22214:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880399c7b400 x1471904511805960/t0(0) o400->atlas2-MDT0000-lwp-OST0227@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss4f8.ccs.ornl.gov kernel: [1914951.456127] Lustre: 22214:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 16:49:57 atlas-oss3f1.ccs.ornl.gov kernel: [1914955.946343] Lustre: 24213:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff88060db5ec00 x1471904508988368/t0(0) o400->atlas2-MDT0000-lwp-OST0268@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3f1.ccs.ornl.gov kernel: [1914956.039230] Lustre: 24213:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:49:57 atlas-oss3b7.ccs.ornl.gov kernel: [1914957.057753] Lustre: 26038:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:49:57 atlas-oss3c6.ccs.ornl.gov kernel: [1914944.389319] Lustre: 24542:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880cfd133800 x1471904508186804/t0(0) o400->atlas2-MDT0000-lwp-OST0135@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3c6.ccs.ornl.gov kernel: [1914944.476849] Lustre: 24542:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:49:57 atlas-oss3b8.ccs.ornl.gov kernel: [1914966.306266] Lustre: 22231:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880c2cbbd800 x1471904516406712/t0(0) o400->atlas2-MDT0000-lwp-OST024f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3b8.ccs.ornl.gov kernel: [1914966.398907] Lustre: 22231:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss3b4.ccs.ornl.gov kernel: [1914963.808851] Lustre: 26922:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff88102bfe6400 x1471904510548480/t0(0) o400->atlas2-MDT0000-lwp-OST009b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3b4.ccs.ornl.gov kernel: [1914963.895459] Lustre: 26922:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:49:57 atlas-oss3i2.ccs.ornl.gov kernel: [1914958.473326] Lustre: 25726:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629630/real 1405629630] req@ffff880cd0649400 x1471904514016196/t0(0) o400->atlas2-MDT0000-lwp-OST0281@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630197 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:49:57 atlas-oss3i2.ccs.ornl.gov kernel: [1914958.560382] Lustre: 25726:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:50:23 atlas-oss3i5.ccs.ornl.gov kernel: [701286.564992] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629943/real 1405629943] req@ffff8805be912400 x1473175721933336/t0(0) o38->atlas2-MDT0000-lwp-OST0164@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630223 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:50:23 atlas-oss3a8.ccs.ornl.gov kernel: [701287.499453] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629943/real 1405629943] req@ffff8802df342400 x1473175720194920/t0(0) o38->atlas2-MDT0000-lwp-OST02d7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630223 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:50:24 atlas-oss3d4.ccs.ornl.gov kernel: [701287.046668] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629943/real 1405629943] req@ffff880870b4bc00 x1473175723812716/t0(0) o38->atlas2-MDT0000-lwp-OST037b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630223 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:50:24 atlas-oss3d4.ccs.ornl.gov kernel: [701287.138700] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:50:43 atlas-oss3i5.ccs.ornl.gov kernel: [701306.187276] Lustre: 17780:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629676/real 1405629676] req@ffff8808708b1400 x1473175721933256/t0(0) o400->atlas2-MDT0000-lwp-OST0044@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630243 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:50:43 atlas-oss3i5.ccs.ornl.gov kernel: [701306.273086] Lustre: 17780:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:50:43 atlas-oss3a8.ccs.ornl.gov kernel: [701307.123672] Lustre: 17741:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629676/real 1405629676] req@ffff88086f1df400 x1473175720194848/t0(0) o400->atlas2-MDT0000-lwp-OST0007@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630243 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:50:43 atlas-oss3a8.ccs.ornl.gov kernel: [701307.209491] Lustre: 17741:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:50:43 atlas-oss3d4.ccs.ornl.gov kernel: [701306.801026] Lustre: 16393:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629676/real 1405629676] req@ffff880dddc7c000 x1473175723812640/t0(0) o400->atlas2-MDT0000-lwp-OST025b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630243 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:50:43 atlas-oss3d4.ccs.ornl.gov kernel: [701306.886109] Lustre: 16393:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:52:26 atlas-oss3f8.ccs.ornl.gov kernel: [801803.411207] Lustre: 81096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880839af2c00 x1473076861320916/t0(0) o400->atlas2-MDT0000-lwp-OST026f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3g5.ccs.ornl.gov kernel: [1913702.599786] Lustre: 13561:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8805a3cdb400 x1471904518249048/t0(0) o400->atlas2-MDT0000-lwp-OST0154@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4f5.ccs.ornl.gov kernel: [1915098.945243] Lustre: 22147:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880a7e190800 x1471904509390904/t0(0) o400->atlas2-MDT0000-lwp-OST0344@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4f5.ccs.ornl.gov kernel: [1915099.037011] Lustre: 22147:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:26 atlas-oss4c2.ccs.ornl.gov kernel: [1915104.957117] Lustre: 25984:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88039e2b8000 x1471904508570636/t0(0) o400->atlas2-MDT0000-lwp-OST0299@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4c2.ccs.ornl.gov kernel: [1915105.048075] Lustre: 25984:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss3f8.ccs.ornl.gov kernel: [801803.504364] Lustre: 81096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:26 atlas-oss3i4.ccs.ornl.gov kernel: [1915106.913792] Lustre: 27280:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880541132800 x1471904510470348/t0(0) o400->atlas2-MDT0000-lwp-OST0283@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3i4.ccs.ornl.gov kernel: [1915107.005514] Lustre: 27280:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss4h8.ccs.ornl.gov kernel: [1915092.010804] Lustre: 24637:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88028a7a0400 x1471904514267164/t0(0) o400->atlas2-MDT0000-lwp-OST01a7@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4h8.ccs.ornl.gov kernel: [1915092.102524] Lustre: 24637:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:26 atlas-oss3g7.ccs.ornl.gov kernel: [1913698.629487] Lustre: 13452:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8801e1486c00 x1471904520422204/t0(0) o400->atlas2-MDT0000-lwp-OST0036@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3g7.ccs.ornl.gov kernel: [1913698.721806] Lustre: 13452:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 16:52:26 atlas-oss3h4.ccs.ornl.gov kernel: [1915104.850898] Lustre: 25741:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff881036343400 x1471904509245240/t0(0) o400->atlas2-MDT0000-lwp-OST039b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3h4.ccs.ornl.gov kernel: [1915104.943483] Lustre: 25741:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss3h1.ccs.ornl.gov kernel: [1915089.790147] Lustre: 22214:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880191cd7000 x1471904517765868/t0(0) o400->atlas2-MDT0000-lwp-OST0278@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4b8.ccs.ornl.gov kernel: [1915102.945199] Lustre: 25800:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8808dc79cc00 x1471904512563872/t0(0) o400->atlas2-MDT0000-lwp-OST0177@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4b8.ccs.ornl.gov kernel: [1915103.036668] Lustre: 25800:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:52:26 atlas-oss3h1.ccs.ornl.gov kernel: [1915089.880598] Lustre: 22214:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss4f4.ccs.ornl.gov kernel: [1915101.584937] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88032195bc00 x1471904516980220/t0(0) o400->atlas2-MDT0000-lwp-OST0193@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4i6.ccs.ornl.gov kernel: [1915114.502537] Lustre: 25729:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88062fdb4800 x1471904510597552/t0(0) o400->atlas2-MDT0000-lwp-OST03ed@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4i6.ccs.ornl.gov kernel: [1915114.595610] Lustre: 25729:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:26 atlas-oss4g3.ccs.ornl.gov kernel: [1915112.264394] Lustre: 25824:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880e87d90800 x1471904517610188/t0(0) o400->atlas2-MDT0000-lwp-OST010a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4g3.ccs.ornl.gov kernel: [1915112.357174] Lustre: 25824:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss4c8.ccs.ornl.gov kernel: [1915092.998107] Lustre: 22049:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880965821c00 x1471904512196896/t0(0) o400->atlas2-MDT0000-lwp-OST03bf@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4c8.ccs.ornl.gov kernel: [1915093.089248] Lustre: 22049:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss4f4.ccs.ornl.gov kernel: [1915101.672404] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:26 atlas-oss4i7.ccs.ornl.gov kernel: [1915114.427511] Lustre: 25151:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880a7838e400 x1471904511375092/t0(0) o400->atlas2-MDT0000-lwp-OST035e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4i7.ccs.ornl.gov kernel: [1915114.518740] Lustre: 25151:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss4d4.ccs.ornl.gov kernel: [1915121.484244] Lustre: 27390:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8803d0c81800 x1471904513679112/t0(0) o400->atlas2-MDT0000-lwp-OST00f3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4d4.ccs.ornl.gov kernel: [1915121.576998] Lustre: 27390:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:52:26 atlas-oss3f6.ccs.ornl.gov kernel: [801805.744570] Lustre: 79675:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880d69d8e400 x1473076856558564/t0(0) o400->atlas2-MDT0000-lwp-OST038d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4g6.ccs.ornl.gov kernel: [1915110.841659] Lustre: 24696:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88031f743800 x1471904507898644/t0(0) o400->atlas2-MDT0000-lwp-OST007d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4g6.ccs.ornl.gov kernel: [1915110.934699] Lustre: 24696:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:52:26 atlas-oss3c1.ccs.ornl.gov kernel: [1915103.298410] Lustre: 23691:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880675e2c800 x1471904511392952/t0(0) o400->atlas2-MDT0000-lwp-OST02e0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3c1.ccs.ornl.gov kernel: [1915103.390867] Lustre: 23691:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss4f1.ccs.ornl.gov kernel: [1915105.351681] Lustre: 25968:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8807900b9000 x1471904513371020/t0(0) o400->atlas2-MDT0000-lwp-OST0100@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4f1.ccs.ornl.gov kernel: [1915105.442398] Lustre: 25968:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss3f6.ccs.ornl.gov kernel: [801805.836126] Lustre: 79675:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss4g7.ccs.ornl.gov kernel: [1915123.491782] Lustre: 26058:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880f9c4c4000 x1471904508873540/t0(0) o400->atlas2-MDT0000-lwp-OST019e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4g7.ccs.ornl.gov kernel: [1915123.583440] Lustre: 26058:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss4b6.ccs.ornl.gov kernel: [1915098.823602] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880124dbe000 x1471904511213544/t0(0) o400->atlas2-MDT0000-lwp-OST03b5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4i8.ccs.ornl.gov kernel: [1915114.901294] Lustre: 26106:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880ff20e7400 x1471904520247088/t0(0) o400->atlas2-MDT0000-lwp-OST01af@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4i8.ccs.ornl.gov kernel: [1915114.994468] Lustre: 26106:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss3b3.ccs.ornl.gov kernel: [1915094.186316] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8806edfc6800 x1471904513820404/t0(0) o400->atlas2-MDT0000-lwp-OST036a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3b3.ccs.ornl.gov kernel: [1915094.277146] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss3e1.ccs.ornl.gov kernel: [1915103.970704] Lustre: 23611:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8802741ce400 x1471904509694324/t0(0) o400->atlas2-MDT0000-lwp-OST02f0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3f5.ccs.ornl.gov kernel: [801793.131265] Lustre: 79202:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8808fc3c7400 x1473076856406272/t0(0) o400->atlas2-MDT0000-lwp-OST038c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3g8.ccs.ornl.gov kernel: [1913699.278695] Lustre: 13443:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8808bc9e3400 x1471904513602396/t0(0) o400->atlas2-MDT0000-lwp-OST0397@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3h7.ccs.ornl.gov kernel: [1915107.958503] Lustre: 25083:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88027b39e000 x1471904510332920/t0(0) o400->atlas2-MDT0000-lwp-OST039e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4b6.ccs.ornl.gov kernel: [1915098.916591] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss3h7.ccs.ornl.gov kernel: [1915108.050321] Lustre: 25083:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss3e1.ccs.ornl.gov kernel: [1915104.061930] Lustre: 23611:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss3f5.ccs.ornl.gov kernel: [801793.221803] Lustre: 79202:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss3g8.ccs.ornl.gov kernel: [1913699.369220] Lustre: 13443:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:26 atlas-oss3a1.ccs.ornl.gov kernel: [1915096.091491] Lustre: 22440:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880d107ca400 x1471904525931740/t0(0) o400->atlas2-MDT0000-lwp-OST0360@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3a1.ccs.ornl.gov kernel: [1915096.181384] Lustre: 22440:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss4g8.ccs.ornl.gov kernel: [1915100.368491] Lustre: 25553:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880308090400 x1471904521746800/t0(0) o400->atlas2-MDT0000-lwp-OST034f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4g8.ccs.ornl.gov kernel: [1915100.460575] Lustre: 25553:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss3b5.ccs.ornl.gov kernel: [1915114.289940] Lustre: 25959:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880692395000 x1471904514716944/t0(0) o400->atlas2-MDT0000-lwp-OST036c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3b5.ccs.ornl.gov kernel: [1915114.381257] Lustre: 25959:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss4b1.ccs.ornl.gov kernel: [1915105.797908] Lustre: 25870:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8803984af800 x1471904507949068/t0(0) o400->atlas2-MDT0000-lwp-OST03b0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4b1.ccs.ornl.gov kernel: [1915105.888365] Lustre: 25870:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss3e7.ccs.ornl.gov kernel: [1915103.710306] Lustre: 25890:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880ddbf5c800 x1471904508195188/t0(0) o400->atlas2-MDT0000-lwp-OST0386@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3e7.ccs.ornl.gov kernel: [1915103.803096] Lustre: 25890:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:26 atlas-oss4f6.ccs.ornl.gov kernel: [1915096.096716] Lustre: 22286:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8808ffde3c00 x1471904511708536/t0(0) o400->atlas2-MDT0000-lwp-OST0345@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4f6.ccs.ornl.gov kernel: [1915096.181273] Lustre: 22286:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:26 atlas-oss3a5.ccs.ornl.gov kernel: [1915094.908558] Lustre: 22234:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88034a2eb000 x1471904511738444/t0(0) o400->atlas2-MDT0000-lwp-OST0364@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3a5.ccs.ornl.gov kernel: [1915095.001558] Lustre: 22234:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:52:26 atlas-oss4d3.ccs.ornl.gov kernel: [1915098.168446] Lustre: 22329:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880cfea72c00 x1471904516978224/t0(0) o400->atlas2-MDT0000-lwp-OST00f2@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4d3.ccs.ornl.gov kernel: [1915098.253906] Lustre: 22329:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:52:26 atlas-oss4i4.ccs.ornl.gov kernel: [1915106.600279] Lustre: 22180:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88102503d800 x1471904516513408/t0(0) o400->atlas2-MDT0000-lwp-OST035b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4i4.ccs.ornl.gov kernel: [1915106.687106] Lustre: 22180:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:52:26 atlas-oss4g5.ccs.ornl.gov kernel: [1915101.818929] Lustre: 26065:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880361a80800 x1471904510456716/t0(0) o400->atlas2-MDT0000-lwp-OST02bc@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4g5.ccs.ornl.gov kernel: [1915101.903241] Lustre: 26065:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:52:26 atlas-oss4h4.ccs.ornl.gov kernel: [1915121.123904] Lustre: 25838:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88063e85b400 x1471904512334324/t0(0) o400->atlas2-MDT0000-lwp-OST01a3@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4h4.ccs.ornl.gov kernel: [1915121.209957] Lustre: 25838:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:26 atlas-oss4e4.ccs.ornl.gov kernel: [1915110.413866] Lustre: 22198:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8806784ad400 x1471904515374292/t0(0) o400->atlas2-MDT0000-lwp-OST018b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4g2.ccs.ornl.gov kernel: [1915112.481967] Lustre: 22142:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88051b9f9c00 x1471904509228956/t0(0) o400->atlas2-MDT0000-lwp-OST03d9@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4e4.ccs.ornl.gov kernel: [1915110.499680] Lustre: 22198:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:52:26 atlas-oss4g2.ccs.ornl.gov kernel: [1915112.572927] Lustre: 22142:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:26 atlas-oss3i7.ccs.ornl.gov kernel: [1915098.878103] Lustre: 24583:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880a634b1400 x1471904508389276/t0(0) o400->atlas2-MDT0000-lwp-OST03a6@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3i7.ccs.ornl.gov kernel: [1915098.962856] Lustre: 24583:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:52:26 atlas-oss4i2.ccs.ornl.gov kernel: [1915124.586622] Lustre: 25752:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8807155a9400 x1471904514634500/t0(0) o400->atlas2-MDT0000-lwp-OST01a9@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4i2.ccs.ornl.gov kernel: [1915124.675772] Lustre: 25752:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:52:26 atlas-oss4f2.ccs.ornl.gov kernel: [1915105.296872] Lustre: 25724:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8806638ca000 x1471904508910652/t0(0) o400->atlas2-MDT0000-lwp-OST0221@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4f2.ccs.ornl.gov kernel: [1915105.383268] Lustre: 25724:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:52:26 atlas-oss4h7.ccs.ornl.gov kernel: [1915110.014599] Lustre: 26119:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88020b36c000 x1471904512615132/t0(0) o400->atlas2-MDT0000-lwp-OST0356@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4h7.ccs.ornl.gov kernel: [1915110.103618] Lustre: 26119:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss4d2.ccs.ornl.gov kernel: [1915106.830862] Lustre: 27431:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880760001000 x1471904512220936/t0(0) o400->atlas2-MDT0000-lwp-OST0211@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4d2.ccs.ornl.gov kernel: [1915106.922024] Lustre: 27431:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:26 atlas-oss4h6.ccs.ornl.gov kernel: [1915100.534425] Lustre: 25969:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8808f834fc00 x1471904515532068/t0(0) o400->atlas2-MDT0000-lwp-OST01a5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4a4.ccs.ornl.gov kernel: [1915103.288361] Lustre: 27357:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8803db488c00 x1471904509591068/t0(0) o400->atlas2-MDT0000-lwp-OST028b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4a4.ccs.ornl.gov kernel: [1915103.378579] Lustre: 27357:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:26 atlas-oss4f7.ccs.ornl.gov kernel: [1915100.811940] Lustre: 25979:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88030065d800 x1471904510455004/t0(0) o400->atlas2-MDT0000-lwp-OST0076@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4f7.ccs.ornl.gov kernel: [1915100.896292] Lustre: 25979:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss4b3.ccs.ornl.gov kernel: [1915096.049854] Lustre: 22310:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8802dfeb7800 x1471904509180504/t0(0) o400->atlas2-MDT0000-lwp-OST03b2@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4b3.ccs.ornl.gov kernel: [1915096.136780] Lustre: 22310:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:26 atlas-oss4c5.ccs.ornl.gov kernel: [1915097.495837] Lustre: 26672:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880183552000 x1471904510171204/t0(0) o400->atlas2-MDT0000-lwp-OST032c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4c5.ccs.ornl.gov kernel: [1915097.589078] Lustre: 26672:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:52:26 atlas-oss4a5.ccs.ornl.gov kernel: [1915120.709407] Lustre: 22135:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880b31623000 x1471904519454112/t0(0) o400->atlas2-MDT0000-lwp-OST01fc@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4a5.ccs.ornl.gov kernel: [1915120.802600] Lustre: 22135:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 16:52:26 atlas-oss4b2.ccs.ornl.gov kernel: [1915124.558461] Lustre: 27474:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88101825f800 x1471904518025816/t0(0) o400->atlas2-MDT0000-lwp-OST0201@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4b2.ccs.ornl.gov kernel: [1915124.647103] Lustre: 27474:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Jul 17 16:52:26 atlas-oss4e7.ccs.ornl.gov kernel: [1915099.168100] Lustre: 26099:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88079858b800 x1471904522621484/t0(0) o400->atlas2-MDT0000-lwp-OST02ae@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4e7.ccs.ornl.gov kernel: [1915099.260096] Lustre: 26099:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 16:52:26 atlas-oss4h2.ccs.ornl.gov kernel: [1915122.978027] Lustre: 26007:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880834cbdc00 x1471904508411112/t0(0) o400->atlas2-MDT0000-lwp-OST0231@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4h2.ccs.ornl.gov kernel: [1915123.069435] Lustre: 26007:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:52:26 atlas-oss3h8.ccs.ornl.gov kernel: [1915098.627811] Lustre: 22191:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880549330c00 x1471904514772792/t0(0) o400->atlas2-MDT0000-lwp-OST027f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3h8.ccs.ornl.gov kernel: [1915098.720873] Lustre: 22191:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:52:26 atlas-oss3d6.ccs.ornl.gov kernel: [1915116.152007] Lustre: 25614:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8810498c1400 x1471904511183336/t0(0) o400->atlas2-MDT0000-lwp-OST001d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3d6.ccs.ornl.gov kernel: [1915116.240958] Lustre: 25614:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:52:26 atlas-oss4h6.ccs.ornl.gov kernel: [1915100.624599] Lustre: 25969:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:26 atlas-oss4a2.ccs.ornl.gov kernel: [1915116.410693] Lustre: 25790:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8804d0b7c400 x1471904510895688/t0(0) o400->atlas2-MDT0000-lwp-OST03a9@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4a2.ccs.ornl.gov kernel: [1915116.496736] Lustre: 25790:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:52:26 atlas-oss3e3.ccs.ornl.gov kernel: [1915098.476923] Lustre: 23072:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8803b3d78c00 x1471904510468420/t0(0) o400->atlas2-MDT0000-lwp-OST01d2@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3e3.ccs.ornl.gov kernel: [1915098.567015] Lustre: 23072:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:52:26 atlas-oss4e2.ccs.ornl.gov kernel: [1915115.588249] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880d9d1e3000 x1471904514146296/t0(0) o400->atlas2-MDT0000-lwp-OST0069@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4e2.ccs.ornl.gov kernel: [1915115.680011] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:52:26 atlas-oss4d7.ccs.ornl.gov kernel: [1915102.495394] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880c1604ac00 x1471904514719960/t0(0) o400->atlas2-MDT0000-lwp-OST0336@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4d7.ccs.ornl.gov kernel: [1915102.583205] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:52:26 atlas-oss4i3.ccs.ornl.gov kernel: [1915094.035490] Lustre: 25959:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8807fe2e7800 x1471904523159548/t0(0) o400->atlas2-MDT0000-lwp-OST01aa@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4i3.ccs.ornl.gov kernel: [1915094.124808] Lustre: 25959:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:52:26 atlas-oss3h5.ccs.ornl.gov kernel: [1915121.520360] Lustre: 24596:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8801ced94c00 x1471904507806548/t0(0) o400->atlas2-MDT0000-lwp-OST00cc@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4e1.ccs.ornl.gov kernel: [1915099.691876] Lustre: 26043:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880f306b2800 x1471904515144720/t0(0) o400->atlas2-MDT0000-lwp-OST0218@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4e1.ccs.ornl.gov kernel: [1915099.783056] Lustre: 26043:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:52:26 atlas-oss3h3.ccs.ornl.gov kernel: [1915097.427755] Lustre: 26634:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880750b4f000 x1471904512413584/t0(0) o400->atlas2-MDT0000-lwp-OST039a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3h3.ccs.ornl.gov kernel: [1915097.518729] Lustre: 26634:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 16:52:26 atlas-oss3f7.ccs.ornl.gov kernel: [801432.910756] Lustre: 78672:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88043e7a0400 x1473076863306156/t0(0) o400->atlas2-MDT0000-lwp-OST01de@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3f7.ccs.ornl.gov kernel: [801432.997036] Lustre: 78672:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:52:26 atlas-oss4c3.ccs.ornl.gov kernel: [1915098.631821] Lustre: 24963:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880ad16e9400 x1471904512246048/t0(0) o400->atlas2-MDT0000-lwp-OST020a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4c3.ccs.ornl.gov kernel: [1915098.718658] Lustre: 24963:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:52:26 atlas-oss3h5.ccs.ornl.gov kernel: [1915121.610888] Lustre: 24596:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:52:26 atlas-oss3h2.ccs.ornl.gov kernel: [1915104.894312] Lustre: 20927:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880ffcd0a000 x1471904518760464/t0(0) o400->atlas2-MDT0000-lwp-OST00c9@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3h2.ccs.ornl.gov kernel: [1915104.984163] Lustre: 20927:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:52:26 atlas-oss4c4.ccs.ornl.gov kernel: [1915113.701804] Lustre: 25789:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880bcc910000 x1471904510981036/t0(0) o400->atlas2-MDT0000-lwp-OST032b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss4c4.ccs.ornl.gov kernel: [1915113.805032] Lustre: 25789:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Jul 17 16:52:26 atlas-oss3f4.ccs.ornl.gov kernel: [1915104.026369] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880cb6f2f000 x1471904507778456/t0(0) o400->atlas2-MDT0000-lwp-OST038b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3f4.ccs.ornl.gov kernel: [1915104.117401] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 19 previous similar messages Jul 17 16:52:26 atlas-oss3a6.ccs.ornl.gov kernel: [1915104.533167] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88042e640800 x1471904512394616/t0(0) o400->atlas2-MDT0000-lwp-OST02d5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3a6.ccs.ornl.gov kernel: [1915104.621809] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:52:26 atlas-oss3e6.ccs.ornl.gov kernel: [1915099.485533] Lustre: 22099:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880fdafe7400 x1471904520534596/t0(0) o400->atlas2-MDT0000-lwp-OST0145@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:26 atlas-oss3e6.ccs.ornl.gov kernel: [1915099.575747] Lustre: 22099:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:52:27 atlas-oss4b7.ccs.ornl.gov kernel: [1915095.989171] Lustre: 23160:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880a1bfc2400 x1471904509377256/t0(0) o400->atlas2-MDT0000-lwp-OST0206@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4b7.ccs.ornl.gov kernel: [1915096.075374] Lustre: 23160:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 16:52:27 atlas-oss4e8.ccs.ornl.gov kernel: [1915114.912101] Lustre: 24598:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff8805b4840800 x1471904518472752/t0(0) o400->atlas2-MDT0000-lwp-OST00ff@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4e8.ccs.ornl.gov kernel: [1915115.005139] Lustre: 24598:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:52:27 atlas-oss4c1.ccs.ornl.gov kernel: [1915107.348317] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff880453bf0400 x1471904516097476/t0(0) o400->atlas2-MDT0000-lwp-OST0298@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4c1.ccs.ornl.gov kernel: [1915107.348323] Lustre: 22135:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629779/real 1405629779] req@ffff88010897d000 x1471904516097460/t0(0) o400->atlas2-MDT0000-lwp-OST0058@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630346 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4c1.ccs.ornl.gov kernel: [1915107.348328] Lustre: 22135:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:27 atlas-oss4c1.ccs.ornl.gov kernel: [1915107.571546] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:27 atlas-oss4g4.ccs.ornl.gov kernel: [1915116.465274] Lustre: 25826:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88075708c000 x1471904515530632/t0(0) o400->atlas2-MDT0000-lwp-OST010b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4a6.ccs.ornl.gov kernel: [1915104.095069] Lustre: 25601:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88018de63000 x1471904512502732/t0(0) o400->atlas2-MDT0000-lwp-OST016d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4a6.ccs.ornl.gov kernel: [1915104.182589] Lustre: 25601:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:52:27 atlas-oss4b5.ccs.ornl.gov kernel: [1915119.590984] Lustre: 25788:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88076226ac00 x1471904518365636/t0(0) o400->atlas2-MDT0000-lwp-OST03b4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4b5.ccs.ornl.gov kernel: [1915119.675855] Lustre: 25788:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss3c7.ccs.ornl.gov kernel: [1915102.921515] Lustre: 25756:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880491c7ec00 x1471904510248644/t0(0) o400->atlas2-MDT0000-lwp-OST0256@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3c7.ccs.ornl.gov kernel: [1915103.009902] Lustre: 25756:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:52:27 atlas-oss3d3.ccs.ornl.gov kernel: [1915108.870349] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88010c202000 x1471904510125676/t0(0) o400->atlas2-MDT0000-lwp-OST00aa@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3d3.ccs.ornl.gov kernel: [1915108.961990] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:52:27 atlas-oss4c6.ccs.ornl.gov kernel: [1915102.964597] Lustre: 22187:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880d14cf7800 x1471904517937176/t0(0) o400->atlas2-MDT0000-lwp-OST005d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4c6.ccs.ornl.gov kernel: [1915103.056246] Lustre: 22187:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss4f3.ccs.ornl.gov kernel: [1915096.781703] Lustre: 26010:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8807b55cb000 x1471904513977104/t0(0) o400->atlas2-MDT0000-lwp-OST0102@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3g2.ccs.ornl.gov kernel: [1913698.687396] Lustre: 14002:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88075711f400 x1471904509036084/t0(0) o400->atlas2-MDT0000-lwp-OST0151@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4f3.ccs.ornl.gov kernel: [1915096.872820] Lustre: 26010:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:27 atlas-oss3g2.ccs.ornl.gov kernel: [1913698.780175] Lustre: 14002:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:52:27 atlas-oss4d6.ccs.ornl.gov kernel: [1915102.149091] Lustre: 25558:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8804097d6800 x1471904519610380/t0(0) o400->atlas2-MDT0000-lwp-OST0215@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4g4.ccs.ornl.gov kernel: [1915116.550793] Lustre: 25826:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:52:27 atlas-oss3e5.ccs.ornl.gov kernel: [1915103.309018] Lustre: 25956:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8807f8f83c00 x1471904510654756/t0(0) o400->atlas2-MDT0000-lwp-OST0024@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4g1.ccs.ornl.gov kernel: [1915099.150008] Lustre: 25820:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8804095c0400 x1471904508970360/t0(0) o400->atlas2-MDT0000-lwp-OST0228@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4a7.ccs.ornl.gov kernel: [1915113.728796] Lustre: 25804:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88060bd08400 x1471904510935988/t0(0) o400->atlas2-MDT0000-lwp-OST01fe@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4a7.ccs.ornl.gov kernel: [1915113.814431] Lustre: 25804:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:52:27 atlas-oss4e3.ccs.ornl.gov kernel: [1915107.886846] Lustre: 27369:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8804367ee000 x1471904515523688/t0(0) o400->atlas2-MDT0000-lwp-OST02aa@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4e3.ccs.ornl.gov kernel: [1915107.971264] Lustre: 27369:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:52:27 atlas-oss3g1.ccs.ornl.gov kernel: [1913692.038248] Lustre: 13846:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880ce3d53c00 x1471904510299592/t0(0) o400->atlas2-MDT0000-lwp-OST0150@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3g1.ccs.ornl.gov kernel: [1913692.125945] Lustre: 13846:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:27 atlas-oss4a1.ccs.ornl.gov kernel: [1915100.023745] Lustre: 25988:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880a6ece8000 x1471904510345780/t0(0) o400->atlas2-MDT0000-lwp-OST03a8@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4a1.ccs.ornl.gov kernel: [1915100.112085] Lustre: 25988:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss3e4.ccs.ornl.gov kernel: [1915116.303315] Lustre: 22101:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880238f11800 x1471904508537984/t0(0) o400->atlas2-MDT0000-lwp-OST0143@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3a2.ccs.ornl.gov kernel: [1915101.610527] Lustre: 25633:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880fd6b66400 x1471904510524252/t0(0) o400->atlas2-MDT0000-lwp-OST02d1@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4e6.ccs.ornl.gov kernel: [1915117.834807] Lustre: 25714:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8801e8b4e800 x1471904508408088/t0(0) o400->atlas2-MDT0000-lwp-OST033d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4e6.ccs.ornl.gov kernel: [1915117.923947] Lustre: 25714:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss4d6.ccs.ornl.gov kernel: [1915102.235036] Lustre: 25558:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:52:27 atlas-oss3e5.ccs.ornl.gov kernel: [1915103.396192] Lustre: 25956:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:52:27 atlas-oss4d5.ccs.ornl.gov kernel: [1915114.595626] Lustre: 26100:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880e3695b400 x1471904519136780/t0(0) o400->atlas2-MDT0000-lwp-OST00f4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4d5.ccs.ornl.gov kernel: [1915114.686838] Lustre: 26100:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss4a3.ccs.ornl.gov kernel: [1915118.052225] Lustre: 28197:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88042b8cf400 x1471904514688852/t0(0) o400->atlas2-MDT0000-lwp-OST016a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4a3.ccs.ornl.gov kernel: [1915118.140164] Lustre: 28197:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss3c8.ccs.ornl.gov kernel: [1915107.147013] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880cdb034000 x1471904507928008/t0(0) o400->atlas2-MDT0000-lwp-OST01c7@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3c8.ccs.ornl.gov kernel: [1915107.232454] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:52:27 atlas-oss3d8.ccs.ornl.gov kernel: [1915108.649020] Lustre: 24576:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880e9da19000 x1471904508710640/t0(0) o400->atlas2-MDT0000-lwp-OST025f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4i5.ccs.ornl.gov kernel: [1915110.050141] Lustre: 25971:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88081c810c00 x1471904512720360/t0(0) o400->atlas2-MDT0000-lwp-OST008c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4i5.ccs.ornl.gov kernel: [1915110.139347] Lustre: 25971:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:52:27 atlas-oss4b4.ccs.ornl.gov kernel: [1915100.222157] Lustre: 26028:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8803c5463c00 x1471904528871968/t0(0) o400->atlas2-MDT0000-lwp-OST0053@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4b4.ccs.ornl.gov kernel: [1915100.308756] Lustre: 26028:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss4g1.ccs.ornl.gov kernel: [1915099.238480] Lustre: 25820:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:52:27 atlas-oss3i8.ccs.ornl.gov kernel: [1915116.001245] Lustre: 22173:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88061ae97c00 x1471904515355432/t0(0) o400->atlas2-MDT0000-lwp-OST0167@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3i8.ccs.ornl.gov kernel: [1915116.088925] Lustre: 22173:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:27 atlas-oss3a2.ccs.ornl.gov kernel: [1915101.700725] Lustre: 25633:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss4h5.ccs.ornl.gov kernel: [1915104.611101] Lustre: 25681:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880422605c00 x1471904509313440/t0(0) o400->atlas2-MDT0000-lwp-OST03e4@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4h5.ccs.ornl.gov kernel: [1915104.700607] Lustre: 25681:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:27 atlas-oss4d1.ccs.ornl.gov kernel: [1915098.136749] Lustre: 22256:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88101f181000 x1471904516811684/t0(0) o400->atlas2-MDT0000-lwp-OST03c0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4d1.ccs.ornl.gov kernel: [1915098.231926] Lustre: 22256:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss3e4.ccs.ornl.gov kernel: [1915116.405438] Lustre: 22101:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:27 atlas-oss3i3.ccs.ornl.gov kernel: [1915097.586360] Lustre: 22142:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880cd2802000 x1471904518169240/t0(0) o400->atlas2-MDT0000-lwp-OST03a2@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3i3.ccs.ornl.gov kernel: [1915097.679749] Lustre: 22142:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss3a4.ccs.ornl.gov kernel: [1915092.108399] Lustre: 22185:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8807f5c86400 x1471904512538744/t0(0) o400->atlas2-MDT0000-lwp-OST0123@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3d2.ccs.ornl.gov kernel: [1915103.829427] Lustre: 25976:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8802f97ab400 x1471904511214600/t0(0) o400->atlas2-MDT0000-lwp-OST0019@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4d8.ccs.ornl.gov kernel: [1915115.169416] Lustre: 22128:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88025d3e5000 x1471904516222956/t0(0) o400->atlas2-MDT0000-lwp-OST0217@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4d8.ccs.ornl.gov kernel: [1915115.259178] Lustre: 22128:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:52:27 atlas-oss3d1.ccs.ornl.gov kernel: [1915094.793579] Lustre: 25671:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8803c2e1ac00 x1471904515778268/t0(0) o400->atlas2-MDT0000-lwp-OST00a8@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3d1.ccs.ornl.gov kernel: [1915094.882306] Lustre: 25671:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Jul 17 16:52:27 atlas-oss3f2.ccs.ornl.gov kernel: [1915097.257551] Lustre: 25717:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880d29ce0000 x1471904513672484/t0(0) o400->atlas2-MDT0000-lwp-OST0149@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3f2.ccs.ornl.gov kernel: [1915097.348893] Lustre: 25717:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss3a3.ccs.ornl.gov kernel: [1915099.365540] Lustre: 27250:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880978d76c00 x1471904526379888/t0(0) o400->atlas2-MDT0000-lwp-OST02d2@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3a3.ccs.ornl.gov kernel: [1915099.451237] Lustre: 27250:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:52:27 atlas-oss3g6.ccs.ornl.gov kernel: [1913710.496229] Lustre: 13983:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8806586d4800 x1471904523473788/t0(0) o400->atlas2-MDT0000-lwp-OST0155@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3g6.ccs.ornl.gov kernel: [1913710.583426] Lustre: 13983:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss3e8.ccs.ornl.gov kernel: [1915102.243686] Lustre: 25797:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880503d3f400 x1471904515139536/t0(0) o400->atlas2-MDT0000-lwp-OST0027@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3e8.ccs.ornl.gov kernel: [1915102.336920] Lustre: 25797:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:27 atlas-oss4a8.ccs.ornl.gov kernel: [1915108.170856] Lustre: 25965:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880b54889400 x1471904507750736/t0(0) o400->atlas2-MDT0000-lwp-OST031f@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4a8.ccs.ornl.gov kernel: [1915108.261347] Lustre: 25965:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:27 atlas-oss3d8.ccs.ornl.gov kernel: [1915108.740770] Lustre: 24576:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:27 atlas-oss4h3.ccs.ornl.gov kernel: [1915114.812598] Lustre: 26071:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88025c063c00 x1471904509650812/t0(0) o400->atlas2-MDT0000-lwp-OST0232@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4h3.ccs.ornl.gov kernel: [1915114.905410] Lustre: 26071:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:52:27 atlas-oss3i6.ccs.ornl.gov kernel: [1915100.989575] Lustre: 25715:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880870903000 x1471904509633976/t0(0) o400->atlas2-MDT0000-lwp-OST0315@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3i6.ccs.ornl.gov kernel: [1915101.081677] Lustre: 25715:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:27 atlas-oss3c3.ccs.ornl.gov kernel: [1915094.822193] Lustre: 22234:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8804d2984400 x1471904511051396/t0(0) o400->atlas2-MDT0000-lwp-OST0012@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3c3.ccs.ornl.gov kernel: [1915094.913170] Lustre: 22234:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:52:27 atlas-oss4h1.ccs.ornl.gov kernel: [1915096.013190] Lustre: 25875:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880cc876cc00 x1471904511603084/t0(0) o400->atlas2-MDT0000-lwp-OST01a0@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3d5.ccs.ornl.gov kernel: [1915105.617730] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88059a5a9800 x1471904509641996/t0(0) o400->atlas2-MDT0000-lwp-OST025c@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3d5.ccs.ornl.gov kernel: [1915105.706076] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:27 atlas-oss4h1.ccs.ornl.gov kernel: [1915096.104743] Lustre: 25875:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:27 atlas-oss3c4.ccs.ornl.gov kernel: [1915097.430073] Lustre: 25954:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880445198c00 x1471904511832884/t0(0) o400->atlas2-MDT0000-lwp-OST0253@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3c4.ccs.ornl.gov kernel: [1915097.514741] Lustre: 25954:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss3b7.ccs.ornl.gov kernel: [1915107.021078] Lustre: 26042:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8802bd9ff000 x1471904519392604/t0(0) o400->atlas2-MDT0000-lwp-OST01be@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3b7.ccs.ornl.gov kernel: [1915107.114039] Lustre: 26042:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:52:27 atlas-oss3a7.ccs.ornl.gov kernel: [1915100.269136] Lustre: 22087:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8804a1a8e400 x1471904518655596/t0(0) o400->atlas2-MDT0000-lwp-OST0246@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3a7.ccs.ornl.gov kernel: [1915100.357208] Lustre: 22087:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss3g3.ccs.ornl.gov kernel: [1913699.526688] Lustre: 13953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880acdfb0800 x1471904516976712/t0(0) o400->atlas2-MDT0000-lwp-OST0032@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4e5.ccs.ornl.gov kernel: [1915103.225177] Lustre: 24647:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8805b696e800 x1471904515279184/t0(0) o400->atlas2-MDT0000-lwp-OST03cc@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4e5.ccs.ornl.gov kernel: [1915103.313258] Lustre: 24647:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 19 previous similar messages Jul 17 16:52:27 atlas-oss4i1.ccs.ornl.gov kernel: [1915102.259549] Lustre: 25782:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88045b6d8000 x1471904513319680/t0(0) o400->atlas2-MDT0000-lwp-OST0088@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3f3.ccs.ornl.gov kernel: [1915115.667948] Lustre: 26022:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880ab4750400 x1471904510688064/t0(0) o400->atlas2-MDT0000-lwp-OST038a@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4i1.ccs.ornl.gov kernel: [1915102.352822] Lustre: 25782:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:27 atlas-oss3f3.ccs.ornl.gov kernel: [1915115.761122] Lustre: 26022:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:27 atlas-oss3b6.ccs.ornl.gov kernel: [1915100.377279] Lustre: 25960:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880517cfb000 x1471904512275356/t0(0) o400->atlas2-MDT0000-lwp-OST036d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3b6.ccs.ornl.gov kernel: [1915100.466187] Lustre: 25960:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:27 atlas-oss3g4.ccs.ornl.gov kernel: [1913712.646563] Lustre: 13618:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88104ca8a400 x1471904510791312/t0(0) o400->atlas2-MDT0000-lwp-OST0033@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3g4.ccs.ornl.gov kernel: [1913712.731797] Lustre: 13618:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:27 atlas-oss3d2.ccs.ornl.gov kernel: [1915103.921835] Lustre: 25976:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:52:27 atlas-oss3i1.ccs.ornl.gov kernel: [1915102.302468] Lustre: 22131:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8805fb384800 x1471904519755364/t0(0) o400->atlas2-MDT0000-lwp-OST0160@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3i1.ccs.ornl.gov kernel: [1915102.388574] Lustre: 22131:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jul 17 16:52:27 atlas-oss3b2.ccs.ornl.gov kernel: [1915102.152805] Lustre: 25279:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880fb5370000 x1471904511916956/t0(0) o400->atlas2-MDT0000-lwp-OST0009@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3b2.ccs.ornl.gov kernel: [1915102.240472] Lustre: 25279:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:52:27 atlas-oss3a4.ccs.ornl.gov kernel: [1915092.196019] Lustre: 22185:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:27 atlas-oss3c2.ccs.ornl.gov kernel: [1915104.214687] Lustre: 25871:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880143455800 x1471904511647900/t0(0) o400->atlas2-MDT0000-lwp-OST01c1@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3f1.ccs.ornl.gov kernel: [1915106.002169] Lustre: 24218:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880208837000 x1471904508988388/t0(0) o400->atlas2-MDT0000-lwp-OST0148@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3d7.ccs.ornl.gov kernel: [1915103.031613] Lustre: 22221:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880e9a490c00 x1471904512149424/t0(0) o400->atlas2-MDT0000-lwp-OST037e@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3f1.ccs.ornl.gov kernel: [1915106.087200] Lustre: 24218:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:52:27 atlas-oss3g3.ccs.ornl.gov kernel: [1913699.614411] Lustre: 13953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss3c5.ccs.ornl.gov kernel: [1915108.777386] Lustre: 25797:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880a201f4800 x1471904512466092/t0(0) o400->atlas2-MDT0000-lwp-OST0254@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3c5.ccs.ornl.gov kernel: [1915108.867949] Lustre: 25797:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:27 atlas-oss4f8.ccs.ornl.gov kernel: [1915101.422081] Lustre: 22224:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880213af1000 x1471904511806000/t0(0) o400->atlas2-MDT0000-lwp-OST0347@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss4f8.ccs.ornl.gov kernel: [1915101.511960] Lustre: 22224:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:27 atlas-oss3c6.ccs.ornl.gov kernel: [1915094.444268] Lustre: 24551:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880ff5c34000 x1471904508186832/t0(0) o400->atlas2-MDT0000-lwp-OST00a5@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3c2.ccs.ornl.gov kernel: [1915104.299939] Lustre: 25871:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:52:27 atlas-oss3d7.ccs.ornl.gov kernel: [1915103.122250] Lustre: 22221:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 16:52:27 atlas-oss3b8.ccs.ornl.gov kernel: [1915116.362779] Lustre: 22225:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880fb24c8c00 x1471904516406748/t0(0) o400->atlas2-MDT0000-lwp-OST02df@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3b8.ccs.ornl.gov kernel: [1915116.455832] Lustre: 22225:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:52:27 atlas-oss3c6.ccs.ornl.gov kernel: [1915094.531801] Lustre: 24551:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:52:27 atlas-oss3e2.ccs.ornl.gov kernel: [1915101.501208] Lustre: 27195:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff88076349c400 x1471904513179156/t0(0) o400->atlas2-MDT0000-lwp-OST01d1@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3e2.ccs.ornl.gov kernel: [1915101.501213] Lustre: 27209:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8807e7a0e000 x1471904513179148/t0(0) o400->atlas2-MDT0000-lwp-OST00b1@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3e2.ccs.ornl.gov kernel: [1915101.501217] Lustre: 27209:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:52:27 atlas-oss3e2.ccs.ornl.gov kernel: [1915101.714999] Lustre: 27195:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:52:27 atlas-oss3h6.ccs.ornl.gov kernel: [1915102.492783] Lustre: 25755:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880c9941bc00 x1471904512216484/t0(0) o400->atlas2-MDT0000-lwp-OST015d@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3h6.ccs.ornl.gov kernel: [1915102.492789] Lustre: 25756:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880fb3c46000 x1471904512216488/t0(0) o400->atlas2-MDT0000-lwp-OST01ed@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3h6.ccs.ornl.gov kernel: [1915102.492794] Lustre: 25756:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:52:27 atlas-oss3b1.ccs.ornl.gov kernel: [1915108.793448] Lustre: 22323:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8804a05a0400 x1471904516110716/t0(0) o400->atlas2-MDT0000-lwp-OST0248@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3b1.ccs.ornl.gov kernel: [1915108.793453] Lustre: 22319:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff8805d585bc00 x1471904516110700/t0(0) o400->atlas2-MDT0000-lwp-OST0008@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3b1.ccs.ornl.gov kernel: [1915108.793458] Lustre: 22319:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:52:27 atlas-oss3h6.ccs.ornl.gov kernel: [1915102.707889] Lustre: 25755:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:52:27 atlas-oss3b4.ccs.ornl.gov kernel: [1915113.865149] Lustre: 26931:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880ef35b1c00 x1471904510548512/t0(0) o400->atlas2-MDT0000-lwp-OST009b@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3b4.ccs.ornl.gov kernel: [1915113.952132] Lustre: 26931:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:52:27 atlas-oss3b1.ccs.ornl.gov kernel: [1915109.013396] Lustre: 22323:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:52:27 atlas-oss3i2.ccs.ornl.gov kernel: [1915108.529258] Lustre: 25718:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629780/real 1405629780] req@ffff880e1c310800 x1471904514016228/t0(0) o400->atlas2-MDT0000-lwp-OST0281@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630347 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:52:27 atlas-oss3i2.ccs.ornl.gov kernel: [1915108.616544] Lustre: 25718:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:53:13 atlas-oss3i5.ccs.ornl.gov kernel: [701456.243066] Lustre: 17777:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629826/real 1405629826] req@ffff880871268c00 x1473175721933304/t0(0) o400->atlas2-MDT0000-lwp-OST0284@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630393 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:53:13 atlas-oss3i5.ccs.ornl.gov kernel: [701456.329765] Lustre: 17777:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:53:13 atlas-oss3a8.ccs.ornl.gov kernel: [701457.182979] Lustre: 17737:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629826/real 1405629826] req@ffff88086f146000 x1473175720194892/t0(0) o400->atlas2-MDT0000-lwp-OST0367@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630393 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:53:13 atlas-oss3a8.ccs.ornl.gov kernel: [701457.275084] Lustre: 17737:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:53:13 atlas-oss3d4.ccs.ornl.gov kernel: [701456.856799] Lustre: 16389:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405629826/real 1405629826] req@ffff880fd7cbfc00 x1473175723812684/t0(0) o400->atlas2-MDT0000-lwp-OST00ab@10.36.226.74@o2ib:12/10 lens 224/224 e 0 to 1 dl 1405630393 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:53:13 atlas-oss3d4.ccs.ornl.gov kernel: [701456.942612] Lustre: 16389:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:54:37 atlas-oss4e4.ccs.ornl.gov kernel: [1915240.704708] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630196/real 1405630196] req@ffff880770c4e800 x1471904515374372/t0(0) o38->atlas2-MDT0000-lwp-OST03cb@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630476 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4e4.ccs.ornl.gov kernel: [1915240.789475] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:54:37 atlas-oss4g2.ccs.ornl.gov kernel: [1915242.787818] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630196/real 1405630196] req@ffff88070636f400 x1471904509229020/t0(0) o38->atlas2-MDT0000-lwp-OST03d9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630476 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4g5.ccs.ornl.gov kernel: [1915232.118892] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630196/real 1405630196] req@ffff8804f2d0dc00 x1471904510456776/t0(0) o38->atlas2-MDT0000-lwp-OST022c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630476 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4g5.ccs.ornl.gov kernel: [1915232.221541] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4h7.ccs.ornl.gov kernel: [1915240.310121] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630196/real 1405630196] req@ffff88025d8f9c00 x1471904512615196/t0(0) o38->atlas2-MDT0000-lwp-OST0356@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630476 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4d2.ccs.ornl.gov kernel: [1915237.129207] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630196/real 1405630196] req@ffff88076eac0800 x1471904512220996/t0(0) o38->atlas2-MDT0000-lwp-OST02a1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630476 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4h6.ccs.ornl.gov kernel: [1915230.838286] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630196/real 1405630196] req@ffff880939ab8400 x1471904515532148/t0(0) o38->atlas2-MDT0000-lwp-OST03e5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630476 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4b3.ccs.ornl.gov kernel: [1915226.350578] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630196/real 1405630196] req@ffff880837c82400 x1471904509180568/t0(0) o38->atlas2-MDT0000-lwp-OST03b2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630476 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4i7.ccs.ornl.gov kernel: [1915245.319838] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630196/real 1405630196] req@ffff880a41c3c400 x1471904511375156/t0(0) o38->atlas2-MDT0000-lwp-OST035e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630476 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4i7.ccs.ornl.gov kernel: [1915245.408829] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4i2.ccs.ornl.gov kernel: [1915254.902336] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630196/real 1405630196] req@ffff88014977c000 x1471904514634580/t0(0) o38->atlas2-MDT0000-lwp-OST03e9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630476 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4f7.ccs.ornl.gov kernel: [1915231.120038] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630196/real 1405630196] req@ffff8806c4d11400 x1471904510455092/t0(0) o38->atlas2-MDT0000-lwp-OST0346@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630476 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4f7.ccs.ornl.gov kernel: [1915231.204911] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4h2.ccs.ornl.gov kernel: [1915253.277549] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630196/real 1405630196] req@ffff8806cb8e3800 x1471904508411172/t0(0) o38->atlas2-MDT0000-lwp-OST02c1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630476 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4h2.ccs.ornl.gov kernel: [1915253.369423] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4b3.ccs.ornl.gov kernel: [1915226.435454] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3e3.ccs.ornl.gov kernel: [1915228.772985] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880717596c00 x1471904510468488/t0(0) o38->atlas2-MDT0000-lwp-OST0262@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4d7.ccs.ornl.gov kernel: [1915232.807857] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880f817d5400 x1471904514720024/t0(0) o38->atlas2-MDT0000-lwp-OST0336@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4i3.ccs.ornl.gov kernel: [1915224.339017] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880ed337d800 x1471904523159628/t0(0) o38->atlas2-MDT0000-lwp-OST03ea@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4i3.ccs.ornl.gov kernel: [1915224.432158] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4i4.ccs.ornl.gov kernel: [1915237.021963] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880b3da32800 x1471904516513468/t0(0) o38->atlas2-MDT0000-lwp-OST02cb@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4i4.ccs.ornl.gov kernel: [1915237.106751] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3h5.ccs.ornl.gov kernel: [1915251.816844] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8804db8b9c00 x1471904507806632/t0(0) o38->atlas2-MDT0000-lwp-OST039c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3c1.ccs.ornl.gov kernel: [1915234.255012] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8804f4553c00 x1471904511393004/t0(0) o38->atlas2-MDT0000-lwp-OST01c0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3f6.ccs.ornl.gov kernel: [801936.702322] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880994705400 x1473076856558628/t0(0) o38->atlas2-MDT0000-lwp-OST038d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3f6.ccs.ornl.gov kernel: [801936.795392] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4f2.ccs.ornl.gov kernel: [1915235.698322] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880e0b51e800 x1471904508910728/t0(0) o38->atlas2-MDT0000-lwp-OST03d1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4f2.ccs.ornl.gov kernel: [1915235.782739] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:54:37 atlas-oss3i7.ccs.ornl.gov kernel: [1915229.277252] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880fe76c4800 x1471904508389328/t0(0) o38->atlas2-MDT0000-lwp-OST01f6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3i7.ccs.ornl.gov kernel: [1915229.370127] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4d7.ccs.ornl.gov kernel: [1915232.892268] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:54:37 atlas-oss4c5.ccs.ornl.gov kernel: [1915227.892709] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880566d13000 x1471904510171256/t0(0) o38->atlas2-MDT0000-lwp-OST020c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4d3.ccs.ornl.gov kernel: [1915228.608515] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880f62312c00 x1471904516978308/t0(0) o38->atlas2-MDT0000-lwp-OST03c2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4c4.ccs.ornl.gov kernel: [1915244.001762] Lustre: 25786:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880be9069000 x1471904510981104/t0(0) o38->atlas2-MDT0000-lwp-OST03bb@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4h4.ccs.ornl.gov kernel: [1915251.549274] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8801a065e400 x1471904512334404/t0(0) o38->atlas2-MDT0000-lwp-OST03e3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4c3.ccs.ornl.gov kernel: [1915228.931744] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880b1c484000 x1471904512246120/t0(0) o38->atlas2-MDT0000-lwp-OST032a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4c3.ccs.ornl.gov kernel: [1915229.017696] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:54:37 atlas-oss3h5.ccs.ornl.gov kernel: [1915251.909941] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3c1.ccs.ornl.gov kernel: [1915234.340251] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss3d6.ccs.ornl.gov kernel: [1915246.545189] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880bc5440400 x1471904511183424/t0(0) o38->atlas2-MDT0000-lwp-OST037d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3d6.ccs.ornl.gov kernel: [1915246.630996] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3g8.ccs.ornl.gov kernel: [1913830.206231] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880d0de82800 x1471904513602460/t0(0) o38->atlas2-MDT0000-lwp-OST0397@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3g8.ccs.ornl.gov kernel: [1913830.298066] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss3h8.ccs.ornl.gov kernel: [1915229.026870] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8808700fb000 x1471904514772864/t0(0) o38->atlas2-MDT0000-lwp-OST039f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3h8.ccs.ornl.gov kernel: [1915229.120051] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3f4.ccs.ornl.gov kernel: [1915234.332535] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880e206d8400 x1471904507778508/t0(0) o38->atlas2-MDT0000-lwp-OST01db@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4a4.ccs.ornl.gov kernel: [1915233.715591] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8804fe04d000 x1471904509591136/t0(0) o38->atlas2-MDT0000-lwp-OST031b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4a4.ccs.ornl.gov kernel: [1915233.807178] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4a5.ccs.ornl.gov kernel: [1915251.129115] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880e19607400 x1471904519454188/t0(0) o38->atlas2-MDT0000-lwp-OST03ac@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4c5.ccs.ornl.gov kernel: [1915227.978118] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:54:37 atlas-oss4a2.ccs.ornl.gov kernel: [1915246.811780] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8806d4735c00 x1471904510895752/t0(0) o38->atlas2-MDT0000-lwp-OST03a9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4c8.ccs.ornl.gov kernel: [1915224.046260] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8808b6439800 x1471904512196960/t0(0) o38->atlas2-MDT0000-lwp-OST03bf@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4e7.ccs.ornl.gov kernel: [1915229.595905] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88023f835c00 x1471904522621556/t0(0) o38->atlas2-MDT0000-lwp-OST03ce@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3e1.ccs.ornl.gov kernel: [1915234.928327] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88077f09e800 x1471904509694388/t0(0) o38->atlas2-MDT0000-lwp-OST02f0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3e1.ccs.ornl.gov kernel: [1915235.030777] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss3f8.ccs.ornl.gov kernel: [801934.589711] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880c88928000 x1473076861320984/t0(0) o38->atlas2-MDT0000-lwp-OST038f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3f8.ccs.ornl.gov kernel: [801934.682836] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss3g5.ccs.ornl.gov kernel: [1913833.786621] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880375a51000 x1471904518249124/t0(0) o38->atlas2-MDT0000-lwp-OST0274@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3f5.ccs.ornl.gov kernel: [801924.087120] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880c80762000 x1473076856406336/t0(0) o38->atlas2-MDT0000-lwp-OST038c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3f5.ccs.ornl.gov kernel: [801924.190022] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4c2.ccs.ornl.gov kernel: [1915236.136650] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880eb9931000 x1471904508570708/t0(0) o38->atlas2-MDT0000-lwp-OST03b9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4c2.ccs.ornl.gov kernel: [1915236.226990] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4a2.ccs.ornl.gov kernel: [1915246.896555] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4c8.ccs.ornl.gov kernel: [1915224.137711] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4e1.ccs.ornl.gov kernel: [1915230.092690] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880fe7bb5400 x1471904515144796/t0(0) o38->atlas2-MDT0000-lwp-OST03c8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3h3.ccs.ornl.gov kernel: [1915227.832791] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880049137000 x1471904512413648/t0(0) o38->atlas2-MDT0000-lwp-OST039a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3a1.ccs.ornl.gov kernel: [1915227.064524] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880a379fb400 x1471904525931804/t0(0) o38->atlas2-MDT0000-lwp-OST0360@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3g5.ccs.ornl.gov kernel: [1913833.871621] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4e2.ccs.ornl.gov kernel: [1915246.045411] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880a47a16400 x1471904514146372/t0(0) o38->atlas2-MDT0000-lwp-OST0219@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4g8.ccs.ornl.gov kernel: [1915231.327861] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880369924c00 x1471904521746856/t0(0) o38->atlas2-MDT0000-lwp-OST022f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4d4.ccs.ornl.gov kernel: [1915252.574001] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880777f14000 x1471904513679196/t0(0) o38->atlas2-MDT0000-lwp-OST03c3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4d4.ccs.ornl.gov kernel: [1915252.666949] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jul 17 16:54:37 atlas-oss3f7.ccs.ornl.gov kernel: [801563.304387] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880734754c00 x1473076863306232/t0(0) o38->atlas2-MDT0000-lwp-OST038e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3h7.ccs.ornl.gov kernel: [1915238.955357] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8803823d6c00 x1471904510332984/t0(0) o38->atlas2-MDT0000-lwp-OST039e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4c1.ccs.ornl.gov kernel: [1915237.746923] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88028c269c00 x1471904516097548/t0(0) o38->atlas2-MDT0000-lwp-OST0328@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4h8.ccs.ornl.gov kernel: [1915223.187981] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880478289000 x1471904514267244/t0(0) o38->atlas2-MDT0000-lwp-OST03e7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4b6.ccs.ornl.gov kernel: [1915229.871643] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880388ca7c00 x1471904511213608/t0(0) o38->atlas2-MDT0000-lwp-OST03b5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4b6.ccs.ornl.gov kernel: [1915229.964693] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4e1.ccs.ornl.gov kernel: [1915230.181817] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4b1.ccs.ornl.gov kernel: [1915236.754633] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88036bfa8800 x1471904507949132/t0(0) o38->atlas2-MDT0000-lwp-OST03b0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4b1.ccs.ornl.gov kernel: [1915236.847355] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss3a1.ccs.ornl.gov kernel: [1915227.150462] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4g8.ccs.ornl.gov kernel: [1915231.419315] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss3h7.ccs.ornl.gov kernel: [1915239.048789] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4h8.ccs.ornl.gov kernel: [1915223.280841] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4f5.ccs.ornl.gov kernel: [1915230.221899] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88102c3d1000 x1471904509390972/t0(0) o38->atlas2-MDT0000-lwp-OST03d4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4f5.ccs.ornl.gov kernel: [1915230.306623] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4g3.ccs.ornl.gov kernel: [1915243.434484] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8809c5c75c00 x1471904517610272/t0(0) o38->atlas2-MDT0000-lwp-OST03da@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3a5.ccs.ornl.gov kernel: [1915225.827740] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88077edaf000 x1471904511738508/t0(0) o38->atlas2-MDT0000-lwp-OST0364@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3a5.ccs.ornl.gov kernel: [1915225.919980] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss3a6.ccs.ornl.gov kernel: [1915234.928532] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88046fb3d000 x1471904512394684/t0(0) o38->atlas2-MDT0000-lwp-OST0365@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3b3.ccs.ornl.gov kernel: [1915225.297294] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8805b2366000 x1471904513820456/t0(0) o38->atlas2-MDT0000-lwp-OST01ba@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3b3.ccs.ornl.gov kernel: [1915225.385373] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss3i4.ccs.ornl.gov kernel: [1915238.200602] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8802a2393c00 x1471904510470420/t0(0) o38->atlas2-MDT0000-lwp-OST03a3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3b5.ccs.ornl.gov kernel: [1915245.339217] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88081165e000 x1471904514717008/t0(0) o38->atlas2-MDT0000-lwp-OST036c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3b5.ccs.ornl.gov kernel: [1915245.430947] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4g3.ccs.ornl.gov kernel: [1915243.527126] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4f6.ccs.ornl.gov kernel: [1915227.108047] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880f78c2f000 x1471904511708600/t0(0) o38->atlas2-MDT0000-lwp-OST0345@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4g6.ccs.ornl.gov kernel: [1915242.019946] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8801dd2fa400 x1471904507898732/t0(0) o38->atlas2-MDT0000-lwp-OST03dd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4g6.ccs.ornl.gov kernel: [1915242.104702] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4g4.ccs.ornl.gov kernel: [1915246.786389] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8806f9526c00 x1471904515530700/t0(0) o38->atlas2-MDT0000-lwp-OST022b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3h4.ccs.ornl.gov kernel: [1915236.128705] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880d5446fc00 x1471904509245304/t0(0) o38->atlas2-MDT0000-lwp-OST039b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3h4.ccs.ornl.gov kernel: [1915236.213165] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss3h2.ccs.ornl.gov kernel: [1915235.389705] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880b04f53000 x1471904518760548/t0(0) o38->atlas2-MDT0000-lwp-OST0399@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3h2.ccs.ornl.gov kernel: [1915235.482545] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4b7.ccs.ornl.gov kernel: [1915226.411277] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880f05c0c800 x1471904509377332/t0(0) o38->atlas2-MDT0000-lwp-OST03b6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4f3.ccs.ornl.gov kernel: [1915227.078310] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8805a3772c00 x1471904513977188/t0(0) o38->atlas2-MDT0000-lwp-OST03d2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4f3.ccs.ornl.gov kernel: [1915227.170781] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3e7.ccs.ornl.gov kernel: [1915234.758834] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880c93c4ec00 x1471904508195252/t0(0) o38->atlas2-MDT0000-lwp-OST0386@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3e7.ccs.ornl.gov kernel: [1915234.843365] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss3g1.ccs.ornl.gov kernel: [1913822.330019] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8803f477a400 x1471904510299664/t0(0) o38->atlas2-MDT0000-lwp-OST01e0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3g1.ccs.ornl.gov kernel: [1913822.414778] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4b8.ccs.ornl.gov kernel: [1915234.212854] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880ccb36f000 x1471904512563944/t0(0) o38->atlas2-MDT0000-lwp-OST0207@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4b8.ccs.ornl.gov kernel: [1915234.306113] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss3i4.ccs.ornl.gov kernel: [1915238.285326] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4f6.ccs.ornl.gov kernel: [1915227.200339] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4g7.ccs.ornl.gov kernel: [1915254.663579] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880aa5afd400 x1471904508873620/t0(0) o38->atlas2-MDT0000-lwp-OST03de@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3g2.ccs.ornl.gov kernel: [1913829.006638] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880253c7b400 x1471904509036164/t0(0) o38->atlas2-MDT0000-lwp-OST0391@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3a2.ccs.ornl.gov kernel: [1915231.905681] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff881068a7bc00 x1471904510524320/t0(0) o38->atlas2-MDT0000-lwp-OST0361@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3h1.ccs.ornl.gov kernel: [1915221.094834] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88025c6fcc00 x1471904517765936/t0(0) o38->atlas2-MDT0000-lwp-OST0308@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4d5.ccs.ornl.gov kernel: [1915244.900471] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8806fd789c00 x1471904519136864/t0(0) o38->atlas2-MDT0000-lwp-OST03c4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4e8.ccs.ornl.gov kernel: [1915245.342943] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8801a7a5c400 x1471904518472836/t0(0) o38->atlas2-MDT0000-lwp-OST03cf@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4i8.ccs.ornl.gov kernel: [1915246.089463] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880813ca7000 x1471904520247168/t0(0) o38->atlas2-MDT0000-lwp-OST03ef@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4i8.ccs.ornl.gov kernel: [1915246.174227] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4e3.ccs.ornl.gov kernel: [1915238.207243] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8806e52aa800 x1471904515523748/t0(0) o38->atlas2-MDT0000-lwp-OST021a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4f1.ccs.ornl.gov kernel: [1915236.569393] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880278428000 x1471904513371104/t0(0) o38->atlas2-MDT0000-lwp-OST03d0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4f1.ccs.ornl.gov kernel: [1915236.661322] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4g7.ccs.ornl.gov kernel: [1915254.753466] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss3i8.ccs.ornl.gov kernel: [1915246.296351] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880465ee4c00 x1471904515355512/t0(0) o38->atlas2-MDT0000-lwp-OST03a7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3i8.ccs.ornl.gov kernel: [1915246.388992] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4i5.ccs.ornl.gov kernel: [1915240.354733] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880465523800 x1471904512720448/t0(0) o38->atlas2-MDT0000-lwp-OST03ec@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4i5.ccs.ornl.gov kernel: [1915240.447812] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Jul 17 16:54:37 atlas-oss3d8.ccs.ornl.gov kernel: [1915238.966788] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880ce8005000 x1471904508710712/t0(0) o38->atlas2-MDT0000-lwp-OST037f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3d8.ccs.ornl.gov kernel: [1915239.060060] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3a4.ccs.ornl.gov kernel: [1915222.408986] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880488a4dc00 x1471904512538824/t0(0) o38->atlas2-MDT0000-lwp-OST0363@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3h1.ccs.ornl.gov kernel: [1915221.188032] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss4h5.ccs.ornl.gov kernel: [1915234.916696] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8804db818000 x1471904509313504/t0(0) o38->atlas2-MDT0000-lwp-OST03e4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3f2.ccs.ornl.gov kernel: [1915227.554376] Lustre: 25710:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88096cf27800 x1471904513672564/t0(0) o38->atlas2-MDT0000-lwp-OST0389@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3f2.ccs.ornl.gov kernel: [1915227.647021] Lustre: 25710:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4c7.ccs.ornl.gov kernel: [1915232.296146] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88039bf80400 x1471904514425656/t0(0) o38->atlas2-MDT0000-lwp-OST03be@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4c7.ccs.ornl.gov kernel: [1915232.381012] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Jul 17 16:54:37 atlas-oss4b5.ccs.ornl.gov kernel: [1915249.985434] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8803836eb800 x1471904518365700/t0(0) o38->atlas2-MDT0000-lwp-OST03b4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4b5.ccs.ornl.gov kernel: [1915250.075329] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4i6.ccs.ornl.gov kernel: [1915245.810930] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8801bdcc1800 x1471904510597620/t0(0) o38->atlas2-MDT0000-lwp-OST035d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4d5.ccs.ornl.gov kernel: [1915244.986860] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:54:37 atlas-oss4e8.ccs.ornl.gov kernel: [1915245.434846] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3d1.ccs.ornl.gov kernel: [1915225.105319] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880823757c00 x1471904515778340/t0(0) o38->atlas2-MDT0000-lwp-OST01c8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3e8.ccs.ornl.gov kernel: [1915232.561683] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880724962c00 x1471904515139624/t0(0) o38->atlas2-MDT0000-lwp-OST0387@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4h1.ccs.ornl.gov kernel: [1915226.319887] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880bad92e400 x1471904511603156/t0(0) o38->atlas2-MDT0000-lwp-OST02c0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3b2.ccs.ornl.gov kernel: [1915232.444994] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880ad881bc00 x1471904511917032/t0(0) o38->atlas2-MDT0000-lwp-OST01b9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3b2.ccs.ornl.gov kernel: [1915232.529555] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3b6.ccs.ornl.gov kernel: [1915230.671315] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8803c61fd800 x1471904512275416/t0(0) o38->atlas2-MDT0000-lwp-OST02dd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3b6.ccs.ornl.gov kernel: [1915230.764212] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:54:37 atlas-oss4f4.ccs.ornl.gov kernel: [1915232.899471] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88045d915c00 x1471904516980288/t0(0) o38->atlas2-MDT0000-lwp-OST0223@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4e5.ccs.ornl.gov kernel: [1915233.528613] Lustre: 24644:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8804c76fb800 x1471904515279236/t0(0) o38->atlas2-MDT0000-lwp-OST021c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3a4.ccs.ornl.gov kernel: [1915222.494811] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3a3.ccs.ornl.gov kernel: [1915229.687340] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880aadd83c00 x1471904526379956/t0(0) o38->atlas2-MDT0000-lwp-OST0362@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4a6.ccs.ornl.gov kernel: [1915234.519478] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88045ee8ac00 x1471904512502812/t0(0) o38->atlas2-MDT0000-lwp-OST03ad@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4a6.ccs.ornl.gov kernel: [1915234.611513] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3e5.ccs.ornl.gov kernel: [1915233.701073] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8805f5f74400 x1471904510654832/t0(0) o38->atlas2-MDT0000-lwp-OST01d4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3e5.ccs.ornl.gov kernel: [1915233.794022] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3g4.ccs.ornl.gov kernel: [1913842.957875] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880e1dff4800 x1471904510791400/t0(0) o38->atlas2-MDT0000-lwp-OST0393@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4g1.ccs.ornl.gov kernel: [1915229.545101] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88037628dc00 x1471904508970436/t0(0) o38->atlas2-MDT0000-lwp-OST03d8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3g4.ccs.ornl.gov kernel: [1913843.051180] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:54:37 atlas-oss4d6.ccs.ornl.gov kernel: [1915232.550342] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8803f45f1c00 x1471904519610448/t0(0) o38->atlas2-MDT0000-lwp-OST02a5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3c2.ccs.ornl.gov kernel: [1915234.515986] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880342599800 x1471904511647976/t0(0) o38->atlas2-MDT0000-lwp-OST0371@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4i6.ccs.ornl.gov kernel: [1915245.896221] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss3c2.ccs.ornl.gov kernel: [1915234.608672] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:54:37 atlas-oss3c5.ccs.ornl.gov kernel: [1915239.073997] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880f14ea8c00 x1471904512466164/t0(0) o38->atlas2-MDT0000-lwp-OST0374@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3f3.ccs.ornl.gov kernel: [1915245.998807] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880068f55400 x1471904510688128/t0(0) o38->atlas2-MDT0000-lwp-OST038a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4f4.ccs.ornl.gov kernel: [1915232.990922] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:54:37 atlas-oss3c6.ccs.ornl.gov kernel: [1915224.739017] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880ff4a40c00 x1471904508186908/t0(0) o38->atlas2-MDT0000-lwp-OST0255@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3c7.ccs.ornl.gov kernel: [1915233.355735] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880767499400 x1471904510248716/t0(0) o38->atlas2-MDT0000-lwp-OST0376@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3c7.ccs.ornl.gov kernel: [1915233.448270] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4a3.ccs.ornl.gov kernel: [1915248.454505] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8805e6774800 x1471904514688932/t0(0) o38->atlas2-MDT0000-lwp-OST03aa@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4a3.ccs.ornl.gov kernel: [1915248.540772] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:54:37 atlas-oss3e4.ccs.ornl.gov kernel: [1915246.733628] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8807ca479800 x1471904508538064/t0(0) o38->atlas2-MDT0000-lwp-OST0383@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3e4.ccs.ornl.gov kernel: [1915246.825279] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4c6.ccs.ornl.gov kernel: [1915233.409960] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff881062535c00 x1471904517937264/t0(0) o38->atlas2-MDT0000-lwp-OST03bd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3c8.ccs.ornl.gov kernel: [1915237.547257] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880ac8fd0400 x1471904507928084/t0(0) o38->atlas2-MDT0000-lwp-OST0377@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3c8.ccs.ornl.gov kernel: [1915237.631420] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:54:37 atlas-oss3d3.ccs.ornl.gov kernel: [1915239.318091] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88083e31dc00 x1471904510125760/t0(0) o38->atlas2-MDT0000-lwp-OST037a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3d3.ccs.ornl.gov kernel: [1915239.411275] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3c5.ccs.ornl.gov kernel: [1915239.167496] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:54:37 atlas-oss4a7.ccs.ornl.gov kernel: [1915244.159804] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8801ed971000 x1471904510936060/t0(0) o38->atlas2-MDT0000-lwp-OST03ae@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4a7.ccs.ornl.gov kernel: [1915244.244658] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:54:37 atlas-oss3e2.ccs.ornl.gov kernel: [1915231.896318] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88082b834400 x1471904513179228/t0(0) o38->atlas2-MDT0000-lwp-OST02f1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4a1.ccs.ornl.gov kernel: [1915230.454663] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880b9104cc00 x1471904510345844/t0(0) o38->atlas2-MDT0000-lwp-OST03a8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4e6.ccs.ornl.gov kernel: [1915248.260098] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88001c7e3800 x1471904508408152/t0(0) o38->atlas2-MDT0000-lwp-OST033d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3h6.ccs.ornl.gov kernel: [1915232.893057] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880fd90d6000 x1471904512216564/t0(0) o38->atlas2-MDT0000-lwp-OST039d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3h6.ccs.ornl.gov kernel: [1915232.986119] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3i6.ccs.ornl.gov kernel: [1915231.386513] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8806ae6a2800 x1471904509634044/t0(0) o38->atlas2-MDT0000-lwp-OST03a5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3i6.ccs.ornl.gov kernel: [1915231.471201] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3d2.ccs.ornl.gov kernel: [1915234.237559] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88093a57c000 x1471904511214688/t0(0) o38->atlas2-MDT0000-lwp-OST0379@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3d2.ccs.ornl.gov kernel: [1915234.330470] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3c4.ccs.ornl.gov kernel: [1915227.820079] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8806cb9bd400 x1471904511832960/t0(0) o38->atlas2-MDT0000-lwp-OST0373@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3c4.ccs.ornl.gov kernel: [1915227.912880] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:54:37 atlas-oss3g3.ccs.ornl.gov kernel: [1913829.921192] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8807a87aa800 x1471904516976800/t0(0) o38->atlas2-MDT0000-lwp-OST0392@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3g3.ccs.ornl.gov kernel: [1913830.013773] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4d1.ccs.ornl.gov kernel: [1915228.566778] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880d6de2c800 x1471904516811748/t0(0) o38->atlas2-MDT0000-lwp-OST03c0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4d1.ccs.ornl.gov kernel: [1915228.659838] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3i3.ccs.ornl.gov kernel: [1915228.015412] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880d61eba400 x1471904518169304/t0(0) o38->atlas2-MDT0000-lwp-OST03a2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3a7.ccs.ornl.gov kernel: [1915230.675129] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8801ea605800 x1471904518655656/t0(0) o38->atlas2-MDT0000-lwp-OST0126@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3g6.ccs.ornl.gov kernel: [1913840.923294] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880f13335400 x1471904523473868/t0(0) o38->atlas2-MDT0000-lwp-OST0395@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3g6.ccs.ornl.gov kernel: [1913841.013341] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4b4.ccs.ornl.gov kernel: [1915230.672961] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff88086522d800 x1471904528872056/t0(0) o38->atlas2-MDT0000-lwp-OST03b3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4h3.ccs.ornl.gov kernel: [1915245.242480] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880fdd9ff000 x1471904509650884/t0(0) o38->atlas2-MDT0000-lwp-OST03e2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3c3.ccs.ornl.gov kernel: [1915225.248056] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8806ee031400 x1471904511051472/t0(0) o38->atlas2-MDT0000-lwp-OST01c2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3d7.ccs.ornl.gov kernel: [1915233.427878] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880e93f83800 x1471904512149480/t0(0) o38->atlas2-MDT0000-lwp-OST025e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3d7.ccs.ornl.gov kernel: [1915233.521328] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:54:37 atlas-oss3f1.ccs.ornl.gov kernel: [1915236.400566] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8804fa596000 x1471904508988468/t0(0) o38->atlas2-MDT0000-lwp-OST0388@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4i1.ccs.ornl.gov kernel: [1915232.678737] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8806dc9aa400 x1471904513319768/t0(0) o38->atlas2-MDT0000-lwp-OST0358@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4i1.ccs.ornl.gov kernel: [1915232.771391] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3e6.ccs.ornl.gov kernel: [1915230.130598] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8809ba928c00 x1471904520534676/t0(0) o38->atlas2-MDT0000-lwp-OST0385@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3e6.ccs.ornl.gov kernel: [1915230.223599] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4a8.ccs.ornl.gov kernel: [1915238.606394] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8809fe7ac000 x1471904507750804/t0(0) o38->atlas2-MDT0000-lwp-OST03af@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4a8.ccs.ornl.gov kernel: [1915238.699555] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4f8.ccs.ornl.gov kernel: [1915231.818196] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880543dc1c00 x1471904511806060/t0(0) o38->atlas2-MDT0000-lwp-OST02b7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3b7.ccs.ornl.gov kernel: [1915237.459462] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8802c7609800 x1471904519392668/t0(0) o38->atlas2-MDT0000-lwp-OST01be@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3b1.ccs.ornl.gov kernel: [1915239.229184] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880807a81c00 x1471904516110788/t0(0) o38->atlas2-MDT0000-lwp-OST02d8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3a7.ccs.ornl.gov kernel: [1915230.766667] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3i1.ccs.ornl.gov kernel: [1915232.733828] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8800556e6800 x1471904519755444/t0(0) o38->atlas2-MDT0000-lwp-OST03a0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3i1.ccs.ornl.gov kernel: [1915232.826866] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss4d8.ccs.ornl.gov kernel: [1915245.646523] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8801494fe400 x1471904516223020/t0(0) o38->atlas2-MDT0000-lwp-OST0217@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss4d8.ccs.ornl.gov kernel: [1915245.738581] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Jul 17 16:54:37 atlas-oss3b7.ccs.ornl.gov kernel: [1915237.552489] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3b8.ccs.ornl.gov kernel: [1915246.773883] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880f0c449800 x1471904516406816/t0(0) o38->atlas2-MDT0000-lwp-OST036f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3b8.ccs.ornl.gov kernel: [1915246.866300] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:54:37 atlas-oss3d5.ccs.ornl.gov kernel: [1915236.131234] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff8805beb53c00 x1471904509642068/t0(0) o38->atlas2-MDT0000-lwp-OST037c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3d5.ccs.ornl.gov kernel: [1915236.224172] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:54:37 atlas-oss3b4.ccs.ornl.gov kernel: [1915244.298115] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff881013d9cc00 x1471904510548596/t0(0) o38->atlas2-MDT0000-lwp-OST036b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3b4.ccs.ornl.gov kernel: [1915244.391389] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:54:37 atlas-oss3i2.ccs.ornl.gov kernel: [1915238.842876] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630197/real 1405630197] req@ffff880343987400 x1471904514016300/t0(0) o38->atlas2-MDT0000-lwp-OST03a1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:54:37 atlas-oss3i2.ccs.ornl.gov kernel: [1915238.936142] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:55:05 atlas-mgs1.ccs.ornl.gov kernel: [795263.484610] Lustre: MGS: haven't heard from client b8d5c572-6703-dc10-338b-e049e892a42a (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff884051392c00, cur 1405630505 expire 1405629605 last 1405629153 Jul 17 16:55:05 atlas-mgs1.ccs.ornl.gov kernel: [795263.508861] Lustre: Skipped 1 previous similar message Jul 17 16:55:23 atlas-oss3i5.ccs.ornl.gov kernel: [701586.677576] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630243/real 1405630243] req@ffff8807a5907400 x1473175721933372/t0(0) o38->atlas2-MDT0000-lwp-OST0314@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630523 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:55:23 atlas-oss3i5.ccs.ornl.gov kernel: [701586.769057] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:55:23 atlas-oss3a8.ccs.ornl.gov kernel: [701587.610128] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630243/real 1405630243] req@ffff88075653c400 x1473175720194956/t0(0) o38->atlas2-MDT0000-lwp-OST0367@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630523 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:55:23 atlas-oss3a8.ccs.ornl.gov kernel: [701587.694862] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 16:55:24 atlas-oss3d4.ccs.ornl.gov kernel: [701587.158260] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630243/real 1405630243] req@ffff8805b1764c00 x1473175723812752/t0(0) o38->atlas2-MDT0000-lwp-OST037b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630523 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:55:24 atlas-oss3d4.ccs.ornl.gov kernel: [701587.251271] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 16:56:39 atlas-oss3e5.ccs.ornl.gov kernel: [1915355.611811] Lustre: atlas2-OST0144: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806c2d1c800, cur 1405630599 expire 1405629699 last 1405629245 Jul 17 16:56:39 atlas-oss3e5.ccs.ornl.gov kernel: [1915355.679671] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss3c7.ccs.ornl.gov kernel: [1915355.331617] Lustre: atlas2-OST0016: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88028f685c00, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss3c2.ccs.ornl.gov kernel: [1915356.510299] Lustre: atlas2-OST00a1: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880870ba7400, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss3b5.ccs.ornl.gov kernel: [1915367.462818] Lustre: atlas2-OST009c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880870859c00, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss3c2.ccs.ornl.gov kernel: [1915356.573833] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss3b5.ccs.ornl.gov kernel: [1915367.526463] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss3e1.ccs.ornl.gov kernel: [1915357.203403] Lustre: atlas2-OST00b0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88081cc13c00, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss3e1.ccs.ornl.gov kernel: [1915357.267099] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss4b1.ccs.ornl.gov kernel: [1915358.947801] Lustre: atlas2-OST0050: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88075aa9f000, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss3i8.ccs.ornl.gov kernel: [1915368.350996] Lustre: atlas2-OST00d7: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8807054f2800, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss4b1.ccs.ornl.gov kernel: [1915359.012748] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss3i8.ccs.ornl.gov kernel: [1915368.414689] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss3d1.ccs.ornl.gov kernel: [1915347.127902] Lustre: atlas2-OST0018: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8807d96bd000, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss3d1.ccs.ornl.gov kernel: [1915347.195609] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss4b6.ccs.ornl.gov kernel: [1915352.072478] Lustre: atlas2-OST0055: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88013a4b9000, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss4a1.ccs.ornl.gov kernel: [1915352.407637] Lustre: atlas2-OST0048: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8808140a5000, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss4a1.ccs.ornl.gov kernel: [1915352.475923] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss3c7.ccs.ornl.gov kernel: [1915355.394781] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss4b6.ccs.ornl.gov kernel: [1915352.140096] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss4d4.ccs.ornl.gov kernel: [1915375.161793] Lustre: atlas2-OST0183: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806c25d5800, cur 1405630599 expire 1405629699 last 1405629247 Jul 17 16:56:39 atlas-oss4d4.ccs.ornl.gov kernel: [1915375.225043] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss4i2.ccs.ornl.gov kernel: [1915377.693996] Lustre: atlas2-OST01a9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880798ea0000, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss4i2.ccs.ornl.gov kernel: [1915377.754018] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss4c6.ccs.ornl.gov kernel: [1915355.747559] Lustre: atlas2-OST00ed: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8804c8078400, cur 1405630599 expire 1405629699 last 1405629247 Jul 17 16:56:39 atlas-oss4c6.ccs.ornl.gov kernel: [1915355.811542] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss3c4.ccs.ornl.gov kernel: [1915350.125788] Lustre: atlas2-OST0133: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806db30f000, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss3c4.ccs.ornl.gov kernel: [1915350.189410] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss3f3.ccs.ornl.gov kernel: [1915368.364679] Lustre: atlas2-OST014a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88025ebd3000, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss3f3.ccs.ornl.gov kernel: [1915368.428316] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss4c4.ccs.ornl.gov kernel: [1915366.695393] Lustre: atlas2-OST017b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8801efb71000, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss3d8.ccs.ornl.gov kernel: [1915361.391051] Lustre: atlas2-OST01cf: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8807dffb5400, cur 1405630599 expire 1405629699 last 1405629245 Jul 17 16:56:39 atlas-oss3d8.ccs.ornl.gov kernel: [1915361.455675] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss4c4.ccs.ornl.gov kernel: [1915366.760590] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss4i6.ccs.ornl.gov kernel: [1915368.215384] Lustre: atlas2-OST01ad: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88071b9fb800, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss4i6.ccs.ornl.gov kernel: [1915368.282120] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss4h6.ccs.ornl.gov kernel: [1915353.630753] Lustre: atlas2-OST01a5: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88083d8a2000, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss4h6.ccs.ornl.gov kernel: [1915353.698052] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss3d6.ccs.ornl.gov kernel: [1915369.218606] Lustre: atlas2-OST013d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88020ee2d000, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss3d6.ccs.ornl.gov kernel: [1915369.286986] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss3h6.ccs.ornl.gov kernel: [1915355.212644] Lustre: atlas2-OST015d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8804f6ca8c00, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss3h6.ccs.ornl.gov kernel: [1915355.281768] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss4e2.ccs.ornl.gov kernel: [1915368.637385] Lustre: atlas2-OST0219: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88081f354400, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss4e2.ccs.ornl.gov kernel: [1915368.705739] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss4c7.ccs.ornl.gov kernel: [1915354.704782] Lustre: atlas2-OST00ee: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88086f02f800, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss4c7.ccs.ornl.gov kernel: [1915354.776308] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss4i1.ccs.ornl.gov kernel: [1915354.955795] Lustre: atlas2-OST01a8: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88086bfe7c00, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss3f5.ccs.ornl.gov kernel: [802046.734546] Lustre: atlas2-OST01dc: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8807e2e17000, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:39 atlas-oss4i1.ccs.ornl.gov kernel: [1915355.026405] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss3f5.ccs.ornl.gov kernel: [802046.804841] Lustre: Skipped 19 previous similar messages Jul 17 16:56:39 atlas-oss3h6.ccs.ornl.gov kernel: [1915355.301587] Lustre: atlas2-OST01ed: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88087030a400, cur 1405630599 expire 1405629699 last 1405629246 Jul 17 16:56:40 atlas-oss3i5.ccs.ornl.gov kernel: [701663.346133] Lustre: atlas2-OST0284: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88106750a400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3i5.ccs.ornl.gov kernel: [701663.408558] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4f4.ccs.ornl.gov kernel: [1915355.982260] Lustre: atlas2-OST0223: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88039e672800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4f4.ccs.ornl.gov kernel: [1915356.045858] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3b5.ccs.ornl.gov kernel: [1915368.643499] Lustre: atlas2-OST012c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805484d5800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3d2.ccs.ornl.gov kernel: [1915357.352531] Lustre: atlas2-OST0139: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8807ebd08800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3d2.ccs.ornl.gov kernel: [1915357.407649] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4h4.ccs.ornl.gov kernel: [1915375.040294] Lustre: atlas2-OST0113: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880754dc5c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4h4.ccs.ornl.gov kernel: [1915375.096507] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4i5.ccs.ornl.gov kernel: [1915363.588765] Lustre: atlas2-OST011c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880669bb5400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3c4.ccs.ornl.gov kernel: [1915350.931426] Lustre: atlas2-OST01c3: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88083c708400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4h7.ccs.ornl.gov kernel: [1915363.914618] Lustre: atlas2-OST0086: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805c62dac00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4h7.ccs.ornl.gov kernel: [1915363.978024] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3i3.ccs.ornl.gov kernel: [1915351.109495] Lustre: atlas2-OST0042: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88081c27ac00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3i3.ccs.ornl.gov kernel: [1915351.173713] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4a6.ccs.ornl.gov kernel: [1915357.702529] Lustre: atlas2-OST004d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880767e42c00, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4a6.ccs.ornl.gov kernel: [1915357.766879] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4b3.ccs.ornl.gov kernel: [1915349.938635] Lustre: atlas2-OST00e2: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880872467c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4b3.ccs.ornl.gov kernel: [1915350.001875] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4g5.ccs.ornl.gov kernel: [1915355.744638] Lustre: atlas2-OST010c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806dd780c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4e5.ccs.ornl.gov kernel: [1915356.723693] Lustre: atlas2-OST00fc: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806af80e400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4c4.ccs.ornl.gov kernel: [1915367.497254] Lustre: atlas2-OST00eb: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880456e0b000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4g5.ccs.ornl.gov kernel: [1915355.807816] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4d3.ccs.ornl.gov kernel: [1915352.105364] Lustre: atlas2-OST00f2: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88031c9e9400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4d4.ccs.ornl.gov kernel: [1915375.970962] Lustre: atlas2-OST00f3: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880870c86000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4e5.ccs.ornl.gov kernel: [1915356.787063] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4d3.ccs.ornl.gov kernel: [1915352.169046] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4d6.ccs.ornl.gov kernel: [1915355.725484] Lustre: atlas2-OST00f5: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806df26e000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4f5.ccs.ornl.gov kernel: [1915353.558271] Lustre: atlas2-OST0104: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8808716c2000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4g1.ccs.ornl.gov kernel: [1915352.719502] Lustre: atlas2-OST0108: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806f5837c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4d6.ccs.ornl.gov kernel: [1915355.789287] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4f5.ccs.ornl.gov kernel: [1915353.621994] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4g1.ccs.ornl.gov kernel: [1915352.783178] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4f8.ccs.ornl.gov kernel: [1915354.891014] Lustre: atlas2-OST0107: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88036a698400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3c2.ccs.ornl.gov kernel: [1915357.691022] Lustre: atlas2-OST0131: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88072214c800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4d2.ccs.ornl.gov kernel: [1915360.731736] Lustre: atlas2-OST00f1: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880758fe0000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3b3.ccs.ornl.gov kernel: [1915348.607912] Lustre: atlas2-OST012a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806e636a400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a7.ccs.ornl.gov kernel: [1915353.769473] Lustre: atlas2-OST0126: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880744671400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3d8.ccs.ornl.gov kernel: [1915362.193512] Lustre: atlas2-OST013f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880507499000, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3f6.ccs.ornl.gov kernel: [802060.218698] Lustre: atlas2-OST014d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8807e314c800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g7.ccs.ornl.gov kernel: [1913953.195296] Lustre: atlas2-OST0156: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8805bb44dc00, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3d3.ccs.ornl.gov kernel: [1915362.463978] Lustre: atlas2-OST013a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8808708f8c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3h4.ccs.ornl.gov kernel: [1915359.415809] Lustre: atlas2-OST015b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806181cbc00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a7.ccs.ornl.gov kernel: [1915353.833360] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3g3.ccs.ornl.gov kernel: [1913953.026826] Lustre: atlas2-OST0152: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8807d9ac4400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3e1.ccs.ornl.gov kernel: [1915358.384159] Lustre: atlas2-OST0140: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8802833c2800, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4b5.ccs.ornl.gov kernel: [1915373.194764] Lustre: atlas2-OST0054: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88024780a800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3c5.ccs.ornl.gov kernel: [1915362.249286] Lustre: atlas2-OST0134: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806ec85b000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4g4.ccs.ornl.gov kernel: [1915370.081627] Lustre: atlas2-OST010b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880753ca1800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3h3.ccs.ornl.gov kernel: [1915351.230667] Lustre: atlas2-OST015a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880126408000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3d3.ccs.ornl.gov kernel: [1915362.527748] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3g7.ccs.ornl.gov kernel: [1913953.258649] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4i4.ccs.ornl.gov kernel: [1915360.538791] Lustre: atlas2-OST011b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880727a00400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3f7.ccs.ornl.gov kernel: [801686.698698] Lustre: atlas2-OST014e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8808152a1400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4b5.ccs.ornl.gov kernel: [1915373.261908] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3g3.ccs.ornl.gov kernel: [1913953.090389] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4g4.ccs.ornl.gov kernel: [1915370.146357] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3h3.ccs.ornl.gov kernel: [1915351.294295] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3f4.ccs.ornl.gov kernel: [1915357.807497] Lustre: atlas2-OST014b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8808704e9400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3f4.ccs.ornl.gov kernel: [1915357.871381] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3c5.ccs.ornl.gov kernel: [1915362.313335] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4i4.ccs.ornl.gov kernel: [1915360.603302] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3f8.ccs.ornl.gov kernel: [802058.035462] Lustre: atlas2-OST014f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8808180f4800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3f8.ccs.ornl.gov kernel: [802058.099552] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3a4.ccs.ornl.gov kernel: [1915345.625368] Lustre: atlas2-OST0123: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880393007400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a4.ccs.ornl.gov kernel: [1915345.690033] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3f7.ccs.ornl.gov kernel: [801686.762502] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4h3.ccs.ornl.gov kernel: [1915368.325432] Lustre: atlas2-OST0112: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880764fe6c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4h3.ccs.ornl.gov kernel: [1915368.390560] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4a7.ccs.ornl.gov kernel: [1915367.297768] Lustre: atlas2-OST016e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880560e56000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4a7.ccs.ornl.gov kernel: [1915367.361388] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4b1.ccs.ornl.gov kernel: [1915360.130116] Lustre: atlas2-OST0170: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8801b7d1d400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3e2.ccs.ornl.gov kernel: [1915355.031059] Lustre: atlas2-OST0141: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880770141c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3e2.ccs.ornl.gov kernel: [1915355.095462] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3i8.ccs.ornl.gov kernel: [1915369.531889] Lustre: atlas2-OST0167: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88054a3da800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4e3.ccs.ornl.gov kernel: [1915361.450060] Lustre: atlas2-OST00fa: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880694bd5c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4e3.ccs.ornl.gov kernel: [1915361.515626] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3a2.ccs.ornl.gov kernel: [1915355.170268] Lustre: atlas2-OST0121: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88073992d400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a2.ccs.ornl.gov kernel: [1915355.235306] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4a4.ccs.ornl.gov kernel: [1915357.188230] Lustre: atlas2-OST016b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880328e6c400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4a4.ccs.ornl.gov kernel: [1915357.252293] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3h7.ccs.ornl.gov kernel: [1915362.351293] Lustre: atlas2-OST015e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88026fcf5800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3h7.ccs.ornl.gov kernel: [1915362.415633] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4d8.ccs.ornl.gov kernel: [1915368.688650] Lustre: atlas2-OST00f7: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806aac1d400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4d8.ccs.ornl.gov kernel: [1915368.754515] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4d7.ccs.ornl.gov kernel: [1915356.332789] Lustre: atlas2-OST00f6: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806bd325c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3h1.ccs.ornl.gov kernel: [1915344.349116] Lustre: atlas2-OST0158: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88081db9e400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4h5.ccs.ornl.gov kernel: [1915358.140079] Lustre: atlas2-OST0114: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88060ea20800, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4h5.ccs.ornl.gov kernel: [1915358.205659] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4c2.ccs.ornl.gov kernel: [1915359.566092] Lustre: atlas2-OST00e9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880514703400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a1.ccs.ornl.gov kernel: [1915350.460494] Lustre: atlas2-OST0120: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880815a99400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4d7.ccs.ornl.gov kernel: [1915356.398771] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4c2.ccs.ornl.gov kernel: [1915359.632226] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4f1.ccs.ornl.gov kernel: [1915359.810807] Lustre: atlas2-OST0100: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8804685d1000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4f1.ccs.ornl.gov kernel: [1915359.876914] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3a1.ccs.ornl.gov kernel: [1915350.526009] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4g3.ccs.ornl.gov kernel: [1915366.776896] Lustre: atlas2-OST010a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880608cf3800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4g3.ccs.ornl.gov kernel: [1915366.842965] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4h6.ccs.ornl.gov kernel: [1915354.432682] Lustre: atlas2-OST0115: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880255c53c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4f6.ccs.ornl.gov kernel: [1915350.409074] Lustre: atlas2-OST0105: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880819c90800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4f6.ccs.ornl.gov kernel: [1915350.475319] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3c3.ccs.ornl.gov kernel: [1915348.330692] Lustre: atlas2-OST0132: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880819f24c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3c3.ccs.ornl.gov kernel: [1915348.396416] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4e1.ccs.ornl.gov kernel: [1915353.500131] Lustre: atlas2-OST00f8: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806a8b75000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4e1.ccs.ornl.gov kernel: [1915353.566675] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3b1.ccs.ornl.gov kernel: [1915362.292456] Lustre: atlas2-OST0128: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88072e0db800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3b1.ccs.ornl.gov kernel: [1915362.358457] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3d6.ccs.ornl.gov kernel: [1915370.024361] Lustre: atlas2-OST01cd: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88087056c000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4c3.ccs.ornl.gov kernel: [1915352.416414] Lustre: atlas2-OST00ea: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88053e38c400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4c3.ccs.ornl.gov kernel: [1915352.483248] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3b4.ccs.ornl.gov kernel: [1915367.210890] Lustre: atlas2-OST012b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88081659dc00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4b2.ccs.ornl.gov kernel: [1915378.438713] Lustre: atlas2-OST0171: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88031fc0e800, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3i6.ccs.ornl.gov kernel: [1915354.500320] Lustre: atlas2-OST0165: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880431a4e400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3e5.ccs.ornl.gov kernel: [1915356.887141] Lustre: atlas2-OST01d4: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806da545000, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3b4.ccs.ornl.gov kernel: [1915367.276943] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4b2.ccs.ornl.gov kernel: [1915378.503725] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3f2.ccs.ornl.gov kernel: [1915350.775848] Lustre: atlas2-OST0149: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805655d6400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4b6.ccs.ornl.gov kernel: [1915353.254728] Lustre: atlas2-OST0175: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880611a29c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3i6.ccs.ornl.gov kernel: [1915354.565790] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3i1.ccs.ornl.gov kernel: [1915355.786938] Lustre: atlas2-OST0160: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805d4f06000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g5.ccs.ornl.gov kernel: [1913957.220594] Lustre: atlas2-OST0154: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88027f133000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3e3.ccs.ornl.gov kernel: [1915352.325014] Lustre: atlas2-OST0142: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880360d66800, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3e7.ccs.ornl.gov kernel: [1915358.032609] Lustre: atlas2-OST0146: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880870721800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3d1.ccs.ornl.gov kernel: [1915348.309800] Lustre: atlas2-OST0138: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88019d9abc00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4d1.ccs.ornl.gov kernel: [1915351.668246] Lustre: atlas2-OST00f0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88031b189000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3f2.ccs.ornl.gov kernel: [1915350.841927] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3i5.ccs.ornl.gov kernel: [701663.461412] Lustre: atlas2-OST0164: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881072e75000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4a5.ccs.ornl.gov kernel: [1915374.594945] Lustre: atlas2-OST016c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8807244d7800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4g2.ccs.ornl.gov kernel: [1915366.401045] Lustre: atlas2-OST0109: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806c3c78000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3i1.ccs.ornl.gov kernel: [1915355.852717] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3d7.ccs.ornl.gov kernel: [1915356.510042] Lustre: atlas2-OST013e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880870ac2c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g5.ccs.ornl.gov kernel: [1913957.286494] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4i2.ccs.ornl.gov kernel: [1915378.495936] Lustre: atlas2-OST0119: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880871cbbc00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3e7.ccs.ornl.gov kernel: [1915358.098893] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3e3.ccs.ornl.gov kernel: [1915352.391259] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4e2.ccs.ornl.gov kernel: [1915369.438240] Lustre: atlas2-OST00f9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806d1409c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4d1.ccs.ornl.gov kernel: [1915351.735778] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4e4.ccs.ornl.gov kernel: [1915364.327460] Lustre: atlas2-OST00fb: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806c0c6a000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g1.ccs.ornl.gov kernel: [1913945.603203] Lustre: atlas2-OST0150: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8801b5791400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4g2.ccs.ornl.gov kernel: [1915366.468319] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4a5.ccs.ornl.gov kernel: [1915374.660703] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3d7.ccs.ornl.gov kernel: [1915356.576485] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4e8.ccs.ornl.gov kernel: [1915368.592513] Lustre: atlas2-OST00ff: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880126a5e000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g1.ccs.ornl.gov kernel: [1913945.669635] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4h1.ccs.ornl.gov kernel: [1915349.519995] Lustre: atlas2-OST0110: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88034b9c0400, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3a8.ccs.ornl.gov kernel: [701664.345024] Lustre: atlas2-OST0127: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8810604c7400, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4e4.ccs.ornl.gov kernel: [1915364.395114] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4f2.ccs.ornl.gov kernel: [1915359.200096] Lustre: atlas2-OST0101: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88087250dc00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3h5.ccs.ornl.gov kernel: [1915375.341163] Lustre: atlas2-OST015c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88033cd4c000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4e8.ccs.ornl.gov kernel: [1915368.660180] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3d5.ccs.ornl.gov kernel: [1915359.123500] Lustre: atlas2-OST013c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806bf6dc000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4a1.ccs.ornl.gov kernel: [1915353.589860] Lustre: atlas2-OST0168: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8804e3b1b000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4i3.ccs.ornl.gov kernel: [1915347.870120] Lustre: atlas2-OST011a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8807225ca800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a8.ccs.ornl.gov kernel: [701664.412270] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4h1.ccs.ornl.gov kernel: [1915349.587652] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4d5.ccs.ornl.gov kernel: [1915368.144760] Lustre: atlas2-OST00f4: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806ee6b4000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3h5.ccs.ornl.gov kernel: [1915375.407742] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4f2.ccs.ornl.gov kernel: [1915359.268086] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3d5.ccs.ornl.gov kernel: [1915359.190652] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4i3.ccs.ornl.gov kernel: [1915347.937833] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4d5.ccs.ornl.gov kernel: [1915368.212998] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4f7.ccs.ornl.gov kernel: [1915354.702377] Lustre: atlas2-OST0106: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88074fa3dc00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4c7.ccs.ornl.gov kernel: [1915355.510006] Lustre: atlas2-OST017e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880754bfe400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4f7.ccs.ornl.gov kernel: [1915354.770722] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4a3.ccs.ornl.gov kernel: [1915371.599958] Lustre: atlas2-OST016a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8802f5837000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4a3.ccs.ornl.gov kernel: [1915371.667134] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4b8.ccs.ornl.gov kernel: [1915357.491872] Lustre: atlas2-OST00e7: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880735986000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4g8.ccs.ornl.gov kernel: [1915354.725842] Lustre: atlas2-OST010f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88061c809800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4i1.ccs.ornl.gov kernel: [1915355.755671] Lustre: atlas2-OST0088: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8801c92a7400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a3.ccs.ornl.gov kernel: [1915352.876880] Lustre: atlas2-OST0122: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806c1260800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3f1.ccs.ornl.gov kernel: [1915359.475657] Lustre: atlas2-OST0148: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8804138b5000, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4g7.ccs.ornl.gov kernel: [1915377.931273] Lustre: atlas2-OST010e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805a3daec00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4b8.ccs.ornl.gov kernel: [1915357.561176] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4e7.ccs.ornl.gov kernel: [1915353.044281] Lustre: atlas2-OST00fe: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880407ba9c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4g8.ccs.ornl.gov kernel: [1915354.794686] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3a3.ccs.ornl.gov kernel: [1915352.945488] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3f5.ccs.ornl.gov kernel: [802047.536950] Lustre: atlas2-OST014c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880839ada800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4h2.ccs.ornl.gov kernel: [1915376.856520] Lustre: atlas2-OST0111: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880870b9b800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4g7.ccs.ornl.gov kernel: [1915378.000270] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3f1.ccs.ornl.gov kernel: [1915359.543731] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4i8.ccs.ornl.gov kernel: [1915369.331699] Lustre: atlas2-OST011f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8804521fd800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3b6.ccs.ornl.gov kernel: [1915353.870688] Lustre: atlas2-OST012d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806f01e0400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4h2.ccs.ornl.gov kernel: [1915376.925723] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4i8.ccs.ornl.gov kernel: [1915369.400656] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3b6.ccs.ornl.gov kernel: [1915353.939405] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4f4.ccs.ornl.gov kernel: [1915356.096060] Lustre: atlas2-OST0103: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88015be0a000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a5.ccs.ornl.gov kernel: [1915349.166475] Lustre: atlas2-OST0124: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880443c23400, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3e4.ccs.ornl.gov kernel: [1915369.882512] Lustre: atlas2-OST0143: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8801a428e400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4c5.ccs.ornl.gov kernel: [1915351.382938] Lustre: atlas2-OST00ec: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880648446c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3c6.ccs.ornl.gov kernel: [1915347.895054] Lustre: atlas2-OST0135: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806e7110400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3f3.ccs.ornl.gov kernel: [1915369.170411] Lustre: atlas2-OST01da: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88032d62f400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3h2.ccs.ornl.gov kernel: [1915358.679489] Lustre: atlas2-OST0159: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88083c0d5000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g8.ccs.ornl.gov kernel: [1913953.684095] Lustre: atlas2-OST0157: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8808705f1800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4h8.ccs.ornl.gov kernel: [1915346.576305] Lustre: atlas2-OST0117: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8801fda6ac00, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3c6.ccs.ornl.gov kernel: [1915347.964596] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3e6.ccs.ornl.gov kernel: [1915353.210470] Lustre: atlas2-OST0145: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806ed12c400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4c5.ccs.ornl.gov kernel: [1915351.453391] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3e4.ccs.ornl.gov kernel: [1915369.951723] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3h2.ccs.ornl.gov kernel: [1915358.748535] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3b8.ccs.ornl.gov kernel: [1915369.792807] Lustre: atlas2-OST012f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88079fe74400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4g6.ccs.ornl.gov kernel: [1915365.310842] Lustre: atlas2-OST010d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8807c0968c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4a2.ccs.ornl.gov kernel: [1915370.264090] Lustre: atlas2-OST0169: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880838748000, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4e6.ccs.ornl.gov kernel: [1915371.389809] Lustre: atlas2-OST00fd: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880272280800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3b7.ccs.ornl.gov kernel: [1915360.528564] Lustre: atlas2-OST012e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8807f5336c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3e6.ccs.ornl.gov kernel: [1915353.279893] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4h8.ccs.ornl.gov kernel: [1915346.646622] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4c6.ccs.ornl.gov kernel: [1915356.552802] Lustre: atlas2-OST017d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8808705ae400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4g6.ccs.ornl.gov kernel: [1915365.381375] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3b8.ccs.ornl.gov kernel: [1915369.862714] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4b7.ccs.ornl.gov kernel: [1915349.698030] Lustre: atlas2-OST00e6: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8805da440400, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3b7.ccs.ornl.gov kernel: [1915360.598575] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4e6.ccs.ornl.gov kernel: [1915371.460630] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4a2.ccs.ornl.gov kernel: [1915370.333258] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4b7.ccs.ornl.gov kernel: [1915349.769186] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3c7.ccs.ornl.gov kernel: [1915356.513495] Lustre: atlas2-OST0136: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8802a4d77000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3h8.ccs.ornl.gov kernel: [1915352.504820] Lustre: atlas2-OST015f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8801325b9800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3h8.ccs.ornl.gov kernel: [1915352.574944] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3c8.ccs.ornl.gov kernel: [1915360.686687] Lustre: atlas2-OST0137: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88070cbce800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4f3.ccs.ornl.gov kernel: [1915350.363918] Lustre: atlas2-OST0102: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805ce607c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3c1.ccs.ornl.gov kernel: [1915357.764706] Lustre: atlas2-OST0130: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880870a13000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a6.ccs.ornl.gov kernel: [1915358.264655] Lustre: atlas2-OST0125: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8807a8247000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3e8.ccs.ornl.gov kernel: [1915355.760422] Lustre: atlas2-OST0147: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805d2279800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3c8.ccs.ornl.gov kernel: [1915360.757697] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4f3.ccs.ornl.gov kernel: [1915350.435820] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3c1.ccs.ornl.gov kernel: [1915357.835796] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4a8.ccs.ornl.gov kernel: [1915361.685709] Lustre: atlas2-OST016f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88073dd9b000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3i7.ccs.ornl.gov kernel: [1915352.785909] Lustre: atlas2-OST0166: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88064f12d800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3e8.ccs.ornl.gov kernel: [1915355.831349] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3a6.ccs.ornl.gov kernel: [1915358.336065] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3g2.ccs.ornl.gov kernel: [1913952.273607] Lustre: atlas2-OST0151: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8805d1f99800, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4a8.ccs.ornl.gov kernel: [1915361.756138] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4b4.ccs.ornl.gov kernel: [1915353.752519] Lustre: atlas2-OST0173: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805cb7cf400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g2.ccs.ornl.gov kernel: [1913952.344557] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3i7.ccs.ornl.gov kernel: [1915352.856647] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4b4.ccs.ornl.gov kernel: [1915353.823053] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4i7.ccs.ornl.gov kernel: [1915368.913148] Lustre: atlas2-OST011e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88074ee88400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3i2.ccs.ornl.gov kernel: [1915361.560336] Lustre: atlas2-OST0161: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8803f6c85c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4i7.ccs.ornl.gov kernel: [1915368.985274] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3i2.ccs.ornl.gov kernel: [1915361.631593] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3g4.ccs.ornl.gov kernel: [1913966.136250] Lustre: atlas2-OST0153: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805d2163c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g4.ccs.ornl.gov kernel: [1913966.208166] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4c8.ccs.ornl.gov kernel: [1915347.498046] Lustre: atlas2-OST00ef: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88039e415c00, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4c8.ccs.ornl.gov kernel: [1915347.572180] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4i6.ccs.ornl.gov kernel: [1915369.017367] Lustre: atlas2-OST011d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88019bf09800, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4f8.ccs.ornl.gov kernel: [1915354.955129] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4d2.ccs.ornl.gov kernel: [1915360.796217] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4c1.ccs.ornl.gov kernel: [1915361.132656] Lustre: atlas2-OST0178: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8801664bc400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3i4.ccs.ornl.gov kernel: [1915361.505862] Lustre: atlas2-OST0163: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88054ef35400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4c1.ccs.ornl.gov kernel: [1915361.207037] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3i4.ccs.ornl.gov kernel: [1915361.580842] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4i5.ccs.ornl.gov kernel: [1915363.645654] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3h1.ccs.ornl.gov kernel: [1915344.413698] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3d4.ccs.ornl.gov kernel: [701663.781272] Lustre: atlas2-OST013b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff881071f56400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3d4.ccs.ornl.gov kernel: [701663.859538] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3b2.ccs.ornl.gov kernel: [1915355.636989] Lustre: atlas2-OST0129: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8808708ba800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3b2.ccs.ornl.gov kernel: [1915355.715843] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3g6.ccs.ornl.gov kernel: [1913964.010879] Lustre: atlas2-OST0155: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805b19ae400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g6.ccs.ornl.gov kernel: [1913964.089611] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss4e7.ccs.ornl.gov kernel: [1915353.113591] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3a5.ccs.ornl.gov kernel: [1915349.236040] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3g8.ccs.ornl.gov kernel: [1913953.753250] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3f6.ccs.ornl.gov kernel: [802060.282127] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3b3.ccs.ornl.gov kernel: [1915348.671663] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3h4.ccs.ornl.gov kernel: [1915359.479134] Lustre: Skipped 19 previous similar messages Jul 17 16:56:40 atlas-oss3d2.ccs.ornl.gov kernel: [1915357.427360] Lustre: atlas2-OST01c9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880430df0000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4h4.ccs.ornl.gov kernel: [1915375.116245] Lustre: atlas2-OST01a3: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880135927400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3e2.ccs.ornl.gov kernel: [1915355.106650] Lustre: atlas2-OST01d1: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880839442c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4i5.ccs.ornl.gov kernel: [1915363.665409] Lustre: atlas2-OST01ac: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88065eac9000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4a4.ccs.ornl.gov kernel: [1915357.263765] Lustre: atlas2-OST01fb: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88081c921c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4f1.ccs.ornl.gov kernel: [1915359.888286] Lustre: atlas2-OST0190: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805f4208c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4f6.ccs.ornl.gov kernel: [1915350.486827] Lustre: atlas2-OST0195: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805ce311400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3b4.ccs.ornl.gov kernel: [1915367.288463] Lustre: atlas2-OST01bb: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8805bca6f000, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3e7.ccs.ornl.gov kernel: [1915358.110223] Lustre: atlas2-OST01d6: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88067d5bb400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g1.ccs.ornl.gov kernel: [1913945.681139] Lustre: atlas2-OST01e0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805dbf65000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4i3.ccs.ornl.gov kernel: [1915347.949170] Lustre: atlas2-OST01aa: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88018b169400, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3h5.ccs.ornl.gov kernel: [1915375.419194] Lustre: atlas2-OST01ec: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880542fea800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4f7.ccs.ornl.gov kernel: [1915354.782329] Lustre: atlas2-OST0196: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88028d083400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3i3.ccs.ornl.gov kernel: [1915351.185271] Lustre: atlas2-OST01f2: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88053a07d400, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4h7.ccs.ornl.gov kernel: [1915363.997835] Lustre: atlas2-OST01a6: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805481c3c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3h2.ccs.ornl.gov kernel: [1915358.759996] Lustre: atlas2-OST01e9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880503ac5000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g8.ccs.ornl.gov kernel: [1913953.764878] Lustre: atlas2-OST01e7: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88086f970000, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4a6.ccs.ornl.gov kernel: [1915357.786581] Lustre: atlas2-OST01fd: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8804c41e5400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4a2.ccs.ornl.gov kernel: [1915370.344301] Lustre: atlas2-OST01f9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8801da52ec00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4g6.ccs.ornl.gov kernel: [1915365.392609] Lustre: atlas2-OST019d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88013a364400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4b3.ccs.ornl.gov kernel: [1915350.021590] Lustre: atlas2-OST0172: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805bf113000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4e5.ccs.ornl.gov kernel: [1915356.798603] Lustre: atlas2-OST018c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8802a9eaac00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4d3.ccs.ornl.gov kernel: [1915352.188950] Lustre: atlas2-OST0182: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88011d370c00, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4g5.ccs.ornl.gov kernel: [1915355.827749] Lustre: atlas2-OST019c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880119cbc400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4g1.ccs.ornl.gov kernel: [1915352.802923] Lustre: atlas2-OST0198: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88086f263c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4f5.ccs.ornl.gov kernel: [1915353.633534] Lustre: atlas2-OST0194: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8804fc786c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4i7.ccs.ornl.gov kernel: [1915368.996771] Lustre: atlas2-OST01ae: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880833610800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4f8.ccs.ornl.gov kernel: [1915354.966721] Lustre: atlas2-OST0197: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8804490c3400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a7.ccs.ornl.gov kernel: [1915353.853233] Lustre: atlas2-OST01b6: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88067d3ba000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3b3.ccs.ornl.gov kernel: [1915348.691525] Lustre: atlas2-OST01ba: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8804e377f400, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4d2.ccs.ornl.gov kernel: [1915360.807866] Lustre: atlas2-OST0181: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8808702f3000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3d3.ccs.ornl.gov kernel: [1915362.547465] Lustre: atlas2-OST01ca: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806f3854000, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3f6.ccs.ornl.gov kernel: [802060.301650] Lustre: atlas2-OST01dd: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8807df9b3800, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3g3.ccs.ornl.gov kernel: [1913953.110150] Lustre: atlas2-OST01e2: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805ac856400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3h4.ccs.ornl.gov kernel: [1915359.499049] Lustre: atlas2-OST01eb: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880596c3b800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g7.ccs.ornl.gov kernel: [1913953.278548] Lustre: atlas2-OST01e6: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8802453ca400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4g4.ccs.ornl.gov kernel: [1915370.157742] Lustre: atlas2-OST019b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88010f5ff000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4b5.ccs.ornl.gov kernel: [1915373.281647] Lustre: atlas2-OST0174: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8804c32d3c00, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4i4.ccs.ornl.gov kernel: [1915360.623122] Lustre: atlas2-OST01ab: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88081a70f000, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3h3.ccs.ornl.gov kernel: [1915351.314022] Lustre: atlas2-OST01ea: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880871cbf000, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3f8.ccs.ornl.gov kernel: [802058.111047] Lustre: atlas2-OST01df: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88081826a000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3c5.ccs.ornl.gov kernel: [1915362.325001] Lustre: atlas2-OST01c4: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806f3785000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4h3.ccs.ornl.gov kernel: [1915368.410444] Lustre: atlas2-OST01a2: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88038fb17800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3f4.ccs.ornl.gov kernel: [1915357.882915] Lustre: atlas2-OST01db: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8805f5fee800, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3f7.ccs.ornl.gov kernel: [801686.782363] Lustre: atlas2-OST01de: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880871821400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a4.ccs.ornl.gov kernel: [1915345.709891] Lustre: atlas2-OST01b3: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880674270400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a2.ccs.ornl.gov kernel: [1915355.254947] Lustre: atlas2-OST01b1: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88072dac0c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4a7.ccs.ornl.gov kernel: [1915367.381141] Lustre: atlas2-OST01fe: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88072324a400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3h7.ccs.ornl.gov kernel: [1915362.427133] Lustre: atlas2-OST01ee: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88083f653400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4c2.ccs.ornl.gov kernel: [1915359.643716] Lustre: atlas2-OST0179: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8808349b1000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4d7.ccs.ornl.gov kernel: [1915356.409748] Lustre: atlas2-OST0186: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806bb9c3c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4e3.ccs.ornl.gov kernel: [1915361.526836] Lustre: atlas2-OST018a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880694bd5000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4h5.ccs.ornl.gov kernel: [1915358.225327] Lustre: atlas2-OST01a4: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88061a077400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4d8.ccs.ornl.gov kernel: [1915368.774406] Lustre: atlas2-OST0187: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806c0600400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3h1.ccs.ornl.gov kernel: [1915344.425019] Lustre: atlas2-OST01e8: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805bf6e3400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a1.ccs.ornl.gov kernel: [1915350.545885] Lustre: atlas2-OST01b0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88087100f400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4g3.ccs.ornl.gov kernel: [1915366.862794] Lustre: atlas2-OST019a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880830e4c800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4e1.ccs.ornl.gov kernel: [1915353.577977] Lustre: atlas2-OST0188: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805edec9800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3c3.ccs.ornl.gov kernel: [1915348.407803] Lustre: atlas2-OST01c2: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880819f41c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4c3.ccs.ornl.gov kernel: [1915352.494703] Lustre: atlas2-OST017a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880730e29000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3b1.ccs.ornl.gov kernel: [1915362.369942] Lustre: atlas2-OST01b8: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880857cf0c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3i6.ccs.ornl.gov kernel: [1915354.577301] Lustre: atlas2-OST01f5: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88086af1b400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4d1.ccs.ornl.gov kernel: [1915351.755417] Lustre: atlas2-OST0180: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88077c028800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g5.ccs.ornl.gov kernel: [1913957.298004] Lustre: atlas2-OST01e4: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880065113000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3e3.ccs.ornl.gov kernel: [1915352.411033] Lustre: atlas2-OST01d2: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880808dcc800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4a5.ccs.ornl.gov kernel: [1915374.672034] Lustre: atlas2-OST01fc: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8807df703000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3f2.ccs.ornl.gov kernel: [1915350.861806] Lustre: atlas2-OST01d9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88083c761800, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4g2.ccs.ornl.gov kernel: [1915366.488195] Lustre: atlas2-OST0199: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806fed03800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3d7.ccs.ornl.gov kernel: [1915356.596337] Lustre: atlas2-OST01ce: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806c2ab1c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4e8.ccs.ornl.gov kernel: [1915368.671578] Lustre: atlas2-OST018f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88075a18d000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4e4.ccs.ornl.gov kernel: [1915364.414994] Lustre: atlas2-OST018b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88043d4e1400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a8.ccs.ornl.gov kernel: [701664.432188] Lustre: atlas2-OST01b7: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8810604dc800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4d5.ccs.ornl.gov kernel: [1915368.224455] Lustre: atlas2-OST0184: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88071d22cc00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4h1.ccs.ornl.gov kernel: [1915349.599174] Lustre: atlas2-OST01a0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88064b4a9000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4f2.ccs.ornl.gov kernel: [1915359.279610] Lustre: atlas2-OST0191: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88028c9c1400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3d5.ccs.ornl.gov kernel: [1915359.201969] Lustre: atlas2-OST01cc: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880870c14c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3f1.ccs.ornl.gov kernel: [1915359.555312] Lustre: atlas2-OST01d8: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880817d07000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4b8.ccs.ornl.gov kernel: [1915357.572673] Lustre: atlas2-OST0177: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8807697ea000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4a3.ccs.ornl.gov kernel: [1915371.687028] Lustre: atlas2-OST01fa: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88075cb43800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a3.ccs.ornl.gov kernel: [1915352.965141] Lustre: atlas2-OST01b2: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806c1260400, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4g8.ccs.ornl.gov kernel: [1915354.814643] Lustre: atlas2-OST019f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805e9aba800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4e7.ccs.ornl.gov kernel: [1915353.125060] Lustre: atlas2-OST018e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880794bc7800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3b6.ccs.ornl.gov kernel: [1915353.950880] Lustre: atlas2-OST01bd: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88087259c400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4i8.ccs.ornl.gov kernel: [1915369.412149] Lustre: atlas2-OST01af: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8801b7933400, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4g7.ccs.ornl.gov kernel: [1915378.020166] Lustre: atlas2-OST019e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805a3dae400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4h2.ccs.ornl.gov kernel: [1915376.937276] Lustre: atlas2-OST01a1: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88044a9ed400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a5.ccs.ornl.gov kernel: [1915349.247260] Lustre: atlas2-OST01b4: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88078af81400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3e4.ccs.ornl.gov kernel: [1915369.971428] Lustre: atlas2-OST01d3: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88033d570000, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3e6.ccs.ornl.gov kernel: [1915353.291270] Lustre: atlas2-OST01d5: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8801bb487800, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3c6.ccs.ornl.gov kernel: [1915347.976034] Lustre: atlas2-OST01c5: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8808372c8800, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4c5.ccs.ornl.gov kernel: [1915351.473300] Lustre: atlas2-OST017c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88033092c800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3b8.ccs.ornl.gov kernel: [1915369.882535] Lustre: atlas2-OST01bf: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880465a58c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4e6.ccs.ornl.gov kernel: [1915371.480509] Lustre: atlas2-OST018d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88070b4f4800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4h8.ccs.ornl.gov kernel: [1915346.666366] Lustre: atlas2-OST01a7: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8808703d7800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3d4.ccs.ornl.gov kernel: [701663.870865] Lustre: atlas2-OST01cb: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880870b2ac00, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss3b7.ccs.ornl.gov kernel: [1915360.610051] Lustre: atlas2-OST01be: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88072e052400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4b7.ccs.ornl.gov kernel: [1915349.780881] Lustre: atlas2-OST0176: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880870983c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3c8.ccs.ornl.gov kernel: [1915360.777573] Lustre: atlas2-OST01c7: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8808701cb800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3h8.ccs.ornl.gov kernel: [1915352.586578] Lustre: atlas2-OST01ef: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880268e35800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3c1.ccs.ornl.gov kernel: [1915357.855721] Lustre: atlas2-OST01c0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88017cce8000, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:40 atlas-oss4f3.ccs.ornl.gov kernel: [1915350.455643] Lustre: atlas2-OST0192: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88086f9d5c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4a8.ccs.ornl.gov kernel: [1915361.767406] Lustre: atlas2-OST01ff: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8807593fcc00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3e8.ccs.ornl.gov kernel: [1915355.851036] Lustre: atlas2-OST01d7: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880419c9c400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3i7.ccs.ornl.gov kernel: [1915352.868137] Lustre: atlas2-OST01f6: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88049b9fe400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g2.ccs.ornl.gov kernel: [1913952.364303] Lustre: atlas2-OST01e1: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8801abee9800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3a6.ccs.ornl.gov kernel: [1915358.347542] Lustre: atlas2-OST01b5: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880870434800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3i2.ccs.ornl.gov kernel: [1915361.643269] Lustre: atlas2-OST01f1: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8804efaba400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g4.ccs.ornl.gov kernel: [1913966.228016] Lustre: atlas2-OST01e3: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88086f123000, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4d6.ccs.ornl.gov kernel: [1915355.809196] Lustre: atlas2-OST0185: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88086fc56c00, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss4c8.ccs.ornl.gov kernel: [1915347.583393] Lustre: atlas2-OST017f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806b9c5b800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3i4.ccs.ornl.gov kernel: [1915361.600544] Lustre: atlas2-OST01f3: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88083bca6800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3i1.ccs.ornl.gov kernel: [1915355.872400] Lustre: atlas2-OST01f0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8805d8edf400, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3b2.ccs.ornl.gov kernel: [1915355.727201] Lustre: atlas2-OST01b9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8806d3911800, cur 1405630600 expire 1405629700 last 1405629248 Jul 17 16:56:40 atlas-oss3g6.ccs.ornl.gov kernel: [1913964.101089] Lustre: atlas2-OST01e5: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806ac93b800, cur 1405630600 expire 1405629700 last 1405629247 Jul 17 16:56:41 atlas-oss3h2.ccs.ornl.gov kernel: [1915359.445874] Lustre: atlas2-OST0039: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880870602400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3g8.ccs.ornl.gov kernel: [1913954.450456] Lustre: atlas2-OST0037: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8804ff527000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3g4.ccs.ornl.gov kernel: [1913966.902633] Lustre: atlas2-OST0033: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8805b7211c00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3g3.ccs.ornl.gov kernel: [1913953.793218] Lustre: atlas2-OST0032: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88086ca62800, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss3g7.ccs.ornl.gov kernel: [1913953.961638] Lustre: atlas2-OST0036: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88082a242000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3h3.ccs.ornl.gov kernel: [1915351.997050] Lustre: atlas2-OST003a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88087289c400, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss3h1.ccs.ornl.gov kernel: [1915345.115466] Lustre: atlas2-OST0038: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88029eec7800, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss3g5.ccs.ornl.gov kernel: [1913957.986963] Lustre: atlas2-OST0034: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880741725000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3h4.ccs.ornl.gov kernel: [1915360.182169] Lustre: atlas2-OST003b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88087139f800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4h8.ccs.ornl.gov kernel: [1915347.540555] Lustre: atlas2-OST02c7: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88065f3c3000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4c3.ccs.ornl.gov kernel: [1915353.386108] Lustre: atlas2-OST005a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8803bf614800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4g6.ccs.ornl.gov kernel: [1915366.566645] Lustre: atlas2-OST007d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806eaa3c400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3b5.ccs.ornl.gov kernel: [1915369.902225] Lustre: atlas2-OST01bc: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880301a65400, cur 1405630601 expire 1405629701 last 1405629249 Jul 17 16:56:41 atlas-oss3d2.ccs.ornl.gov kernel: [1915358.608159] Lustre: atlas2-OST00a9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88034fb68c00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4h4.ccs.ornl.gov kernel: [1915376.296094] Lustre: atlas2-OST0083: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88018b4d6400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4c1.ccs.ornl.gov kernel: [1915362.386332] Lustre: atlas2-OST0058: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880641c66c00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3h5.ccs.ornl.gov kernel: [1915376.594964] Lustre: atlas2-OST003c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806d3596000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4b2.ccs.ornl.gov kernel: [1915379.692369] Lustre: atlas2-OST0051: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880752017800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3i1.ccs.ornl.gov kernel: [1915357.040648] Lustre: atlas2-OST0040: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8805d6197400, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss3i6.ccs.ornl.gov kernel: [1915355.754101] Lustre: atlas2-OST0045: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880448918000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3h6.ccs.ornl.gov kernel: [1915357.270141] Lustre: atlas2-OST003d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880250e3d800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4a5.ccs.ornl.gov kernel: [1915375.848676] Lustre: atlas2-OST004c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8802f84a4800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4c7.ccs.ornl.gov kernel: [1915356.763751] Lustre: atlas2-OST005e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8807536a3800, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4d5.ccs.ornl.gov kernel: [1915369.400573] Lustre: atlas2-OST0064: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8803cf52e000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3i5.ccs.ornl.gov kernel: [701664.715138] Lustre: atlas2-OST0044: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8807ccf64800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4e8.ccs.ornl.gov kernel: [1915369.848267] Lustre: atlas2-OST006f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8804b01eac00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4e4.ccs.ornl.gov kernel: [1915365.583209] Lustre: atlas2-OST006b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806bd1dc400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4a3.ccs.ornl.gov kernel: [1915372.853641] Lustre: atlas2-OST004a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880752a32000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3i5.ccs.ornl.gov kernel: [701664.778955] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss4h1.ccs.ornl.gov kernel: [1915350.775801] Lustre: atlas2-OST0080: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880871196c00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4f7.ccs.ornl.gov kernel: [1915355.958162] Lustre: atlas2-OST0076: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880249b91400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4e2.ccs.ornl.gov kernel: [1915370.694028] Lustre: atlas2-OST0069: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806d280e000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4f2.ccs.ornl.gov kernel: [1915360.455854] Lustre: atlas2-OST0071: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880686282400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4g8.ccs.ornl.gov kernel: [1915355.981588] Lustre: atlas2-OST007f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880779ed8400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4g7.ccs.ornl.gov kernel: [1915379.187038] Lustre: atlas2-OST007e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88069e0ac400, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4b8.ccs.ornl.gov kernel: [1915358.747639] Lustre: atlas2-OST0057: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8805986c2000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4e2.ccs.ornl.gov kernel: [1915370.757817] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss3b6.ccs.ornl.gov kernel: [1915355.126279] Lustre: atlas2-OST009d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8802857acc00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4e7.ccs.ornl.gov kernel: [1915354.300065] Lustre: atlas2-OST006e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880354eee800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3d4.ccs.ornl.gov kernel: [701665.036870] Lustre: atlas2-OST00ab: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88078ea9cc00, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss3a3.ccs.ornl.gov kernel: [1915354.132510] Lustre: atlas2-OST0092: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880838af9800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4i3.ccs.ornl.gov kernel: [1915349.125913] Lustre: atlas2-OST008a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880721cec800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4i8.ccs.ornl.gov kernel: [1915370.587378] Lustre: atlas2-OST008f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8807609ff800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3b2.ccs.ornl.gov kernel: [1915356.892645] Lustre: atlas2-OST0099: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880490802000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4h2.ccs.ornl.gov kernel: [1915378.112262] Lustre: atlas2-OST0081: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880870621800, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss3f1.ccs.ornl.gov kernel: [1915360.731261] Lustre: atlas2-OST00b8: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880558665400, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4f4.ccs.ornl.gov kernel: [1915357.351834] Lustre: atlas2-OST0073: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8804b909d000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3g6.ccs.ornl.gov kernel: [1913965.266486] Lustre: atlas2-OST00c5: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88032740ec00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3f5.ccs.ornl.gov kernel: [802048.792492] Lustre: atlas2-OST00bc: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8808711bac00, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss3c4.ccs.ornl.gov kernel: [1915352.184939] Lustre: atlas2-OST00a3: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88014858c800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3f3.ccs.ornl.gov kernel: [1915370.423902] Lustre: atlas2-OST00ba: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8805c7312000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4f4.ccs.ornl.gov kernel: [1915357.416348] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss3e4.ccs.ornl.gov kernel: [1915371.138110] Lustre: atlas2-OST00b3: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8807223e9000, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss3a5.ccs.ornl.gov kernel: [1915350.422149] Lustre: atlas2-OST0094: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8804a7d8f800, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4c5.ccs.ornl.gov kernel: [1915352.638721] Lustre: atlas2-OST005c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8808160ff000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3e6.ccs.ornl.gov kernel: [1915354.466069] Lustre: atlas2-OST00b5: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8807a5eff400, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss3c6.ccs.ornl.gov kernel: [1915349.150735] Lustre: atlas2-OST00a5: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806e00cb000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3e6.ccs.ornl.gov kernel: [1915354.530195] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss4e6.ccs.ornl.gov kernel: [1915372.645590] Lustre: atlas2-OST006d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880724c57c00, cur 1405630601 expire 1405629701 last 1405629249 Jul 17 16:56:41 atlas-oss3b8.ccs.ornl.gov kernel: [1915371.048453] Lustre: atlas2-OST009f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806efaf4000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4a2.ccs.ornl.gov kernel: [1915371.517758] Lustre: atlas2-OST0049: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880509752000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3b7.ccs.ornl.gov kernel: [1915361.784196] Lustre: atlas2-OST009e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8807f4fa1c00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4b7.ccs.ornl.gov kernel: [1915350.953773] Lustre: atlas2-OST0056: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880871d09c00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4c6.ccs.ornl.gov kernel: [1915357.806505] Lustre: atlas2-OST005d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88021b27b000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4a6.ccs.ornl.gov kernel: [1915358.962294] Lustre: atlas2-OST00dd: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88074e2b7000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3h8.ccs.ornl.gov kernel: [1915353.758590] Lustre: atlas2-OST003f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8805c8578c00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3i3.ccs.ornl.gov kernel: [1915352.369204] Lustre: atlas2-OST00d2: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88081b15fc00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4f3.ccs.ornl.gov kernel: [1915351.619694] Lustre: atlas2-OST0072: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88059dc0a400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3c7.ccs.ornl.gov kernel: [1915357.769107] Lustre: atlas2-OST00a6: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8802a4d76c00, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4a6.ccs.ornl.gov kernel: [1915359.026929] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss3i3.ccs.ornl.gov kernel: [1915352.434161] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss3c1.ccs.ornl.gov kernel: [1915359.020364] Lustre: atlas2-OST00a0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88081e141800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3g2.ccs.ornl.gov kernel: [1913953.529174] Lustre: atlas2-OST00c1: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880194b49800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3c7.ccs.ornl.gov kernel: [1915357.834653] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss4a8.ccs.ornl.gov kernel: [1915362.939398] Lustre: atlas2-OST004f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880792652400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3e8.ccs.ornl.gov kernel: [1915357.016000] Lustre: atlas2-OST00b7: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88085d19b000, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss3a6.ccs.ornl.gov kernel: [1915359.520468] Lustre: atlas2-OST0095: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880730aad800, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4b3.ccs.ornl.gov kernel: [1915351.194382] Lustre: atlas2-OST0052: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8808728d8000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3c8.ccs.ornl.gov kernel: [1915361.942329] Lustre: atlas2-OST00a7: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880614b56400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4g5.ccs.ornl.gov kernel: [1915357.000424] Lustre: atlas2-OST007c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880870e3dc00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3i7.ccs.ornl.gov kernel: [1915354.039671] Lustre: atlas2-OST0046: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880872dd1c00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4b4.ccs.ornl.gov kernel: [1915355.006112] Lustre: atlas2-OST0053: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8805e5723800, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4d3.ccs.ornl.gov kernel: [1915353.361136] Lustre: atlas2-OST0062: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880711eef800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4g1.ccs.ornl.gov kernel: [1915353.975279] Lustre: atlas2-OST0078: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880586057400, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4d3.ccs.ornl.gov kernel: [1915353.428414] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss4g1.ccs.ornl.gov kernel: [1915354.042404] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss4i7.ccs.ornl.gov kernel: [1915370.168823] Lustre: atlas2-OST008e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88083daebc00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3b3.ccs.ornl.gov kernel: [1915349.863618] Lustre: atlas2-OST009a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88049a7fd400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3a7.ccs.ornl.gov kernel: [1915355.025237] Lustre: atlas2-OST0096: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88087125c400, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss3i2.ccs.ornl.gov kernel: [1915362.814092] Lustre: atlas2-OST0041: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8803f6c85000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4d4.ccs.ornl.gov kernel: [1915377.226734] Lustre: atlas2-OST0063: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880871be3c00, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4e5.ccs.ornl.gov kernel: [1915357.979468] Lustre: atlas2-OST006c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880663c14000, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4c4.ccs.ornl.gov kernel: [1915368.753069] Lustre: atlas2-OST005b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88026e6c4800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3d8.ccs.ornl.gov kernel: [1915363.449055] Lustre: atlas2-OST00af: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88076e916400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3c2.ccs.ornl.gov kernel: [1915358.950951] Lustre: atlas2-OST0251: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880664d63400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3c2.ccs.ornl.gov kernel: [1915359.014375] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss4d6.ccs.ornl.gov kernel: [1915356.981266] Lustre: atlas2-OST0065: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806e13e9000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4f5.ccs.ornl.gov kernel: [1915354.814039] Lustre: atlas2-OST0074: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8805a11a1000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3e1.ccs.ornl.gov kernel: [1915359.644049] Lustre: atlas2-OST0260: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88053c743c00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3e1.ccs.ornl.gov kernel: [1915359.707447] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss4c8.ccs.ornl.gov kernel: [1915348.753797] Lustre: atlas2-OST005f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880305d08000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4f8.ccs.ornl.gov kernel: [1915356.146804] Lustre: atlas2-OST0077: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880583613800, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4a7.ccs.ornl.gov kernel: [1915368.551462] Lustre: atlas2-OST004e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88072b92d000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4b5.ccs.ornl.gov kernel: [1915374.454541] Lustre: atlas2-OST00e4: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8805aa5d6c00, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4d2.ccs.ornl.gov kernel: [1915361.987531] Lustre: atlas2-OST0061: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88072d9bb000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3f7.ccs.ornl.gov kernel: [801687.954397] Lustre: atlas2-OST00be: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8807e5830400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4b1.ccs.ornl.gov kernel: [1915361.385678] Lustre: atlas2-OST00e0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880816cb0c00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4b1.ccs.ornl.gov kernel: [1915361.453628] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss4g4.ccs.ornl.gov kernel: [1915371.337463] Lustre: atlas2-OST007b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880557812800, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4i4.ccs.ornl.gov kernel: [1915361.794550] Lustre: atlas2-OST008b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88073bf6e800, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss3f4.ccs.ornl.gov kernel: [1915359.063054] Lustre: atlas2-OST00bb: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8804f1795000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3c5.ccs.ornl.gov kernel: [1915363.504944] Lustre: atlas2-OST00a4: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880819e74000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3a4.ccs.ornl.gov kernel: [1915346.880995] Lustre: atlas2-OST0093: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88083d8f8000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3f8.ccs.ornl.gov kernel: [802059.291038] Lustre: atlas2-OST00bf: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88083d55fc00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3i8.ccs.ornl.gov kernel: [1915370.785579] Lustre: atlas2-OST0047: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88024cd95000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3a2.ccs.ornl.gov kernel: [1915356.425952] Lustre: atlas2-OST0091: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880542492400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3i8.ccs.ornl.gov kernel: [1915370.856044] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss4i6.ccs.ornl.gov kernel: [1915370.273138] Lustre: atlas2-OST008d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8804ad9b3000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4h3.ccs.ornl.gov kernel: [1915369.581205] Lustre: atlas2-OST0082: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880646007000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4d7.ccs.ornl.gov kernel: [1915357.588553] Lustre: atlas2-OST0066: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880732a28000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3e2.ccs.ornl.gov kernel: [1915356.286589] Lustre: atlas2-OST00b1: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880283b2e800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4e3.ccs.ornl.gov kernel: [1915362.705783] Lustre: atlas2-OST006a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806a9252c00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4h5.ccs.ornl.gov kernel: [1915359.395839] Lustre: atlas2-OST0084: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880607f80000, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4i5.ccs.ornl.gov kernel: [1915364.844521] Lustre: atlas2-OST008c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8808727d0800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3a1.ccs.ornl.gov kernel: [1915351.716178] Lustre: atlas2-OST0090: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88034bd5f400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4d8.ccs.ornl.gov kernel: [1915369.944405] Lustre: atlas2-OST0067: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880871170000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4a4.ccs.ornl.gov kernel: [1915358.441914] Lustre: atlas2-OST004b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8801241aec00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4c2.ccs.ornl.gov kernel: [1915360.821884] Lustre: atlas2-OST0059: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88086c4dec00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4g3.ccs.ornl.gov kernel: [1915368.032681] Lustre: atlas2-OST007a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8808533cc000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4f1.ccs.ornl.gov kernel: [1915361.066596] Lustre: atlas2-OST0070: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88076f3e6400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4h6.ccs.ornl.gov kernel: [1915355.688506] Lustre: atlas2-OST0085: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8801ee1b4000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3i4.ccs.ornl.gov kernel: [1915362.759656] Lustre: atlas2-OST0043: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880548792c00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3e5.ccs.ornl.gov kernel: [1915358.140504] Lustre: atlas2-OST00b4: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8808384e5400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4e1.ccs.ornl.gov kernel: [1915354.755930] Lustre: atlas2-OST0068: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806a9045800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3d6.ccs.ornl.gov kernel: [1915371.277720] Lustre: atlas2-OST00ad: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806e53e9400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3b1.ccs.ornl.gov kernel: [1915363.548185] Lustre: atlas2-OST0098: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8808711c2c00, cur 1405630601 expire 1405629701 last 1405629249 Jul 17 16:56:41 atlas-oss4f6.ccs.ornl.gov kernel: [1915351.664858] Lustre: atlas2-OST0075: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8805dc960800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3c3.ccs.ornl.gov kernel: [1915349.586315] Lustre: atlas2-OST00a2: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88081a519800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3b4.ccs.ornl.gov kernel: [1915368.466560] Lustre: atlas2-OST009b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880813e6d000, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4d1.ccs.ornl.gov kernel: [1915352.924065] Lustre: atlas2-OST0060: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88081905f400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3f2.ccs.ornl.gov kernel: [1915352.031458] Lustre: atlas2-OST00b9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88086ca7ac00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3e3.ccs.ornl.gov kernel: [1915353.580579] Lustre: atlas2-OST00b2: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8805cf0d3c00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3d1.ccs.ornl.gov kernel: [1915349.565422] Lustre: atlas2-OST00a8: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8802f2a34000, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss3d7.ccs.ornl.gov kernel: [1915357.765586] Lustre: atlas2-OST00ae: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806514c4000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4b6.ccs.ornl.gov kernel: [1915354.510282] Lustre: atlas2-OST00e5: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8805b479e400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3e7.ccs.ornl.gov kernel: [1915359.288206] Lustre: atlas2-OST00b6: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88056fcab400, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4i2.ccs.ornl.gov kernel: [1915379.751696] Lustre: atlas2-OST0089: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88072bb5e000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4g2.ccs.ornl.gov kernel: [1915367.656836] Lustre: atlas2-OST0079: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8804d28cbc00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3d1.ccs.ornl.gov kernel: [1915349.636705] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss3g1.ccs.ornl.gov kernel: [1913946.858808] Lustre: atlas2-OST00c0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806076ed400, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss3a8.ccs.ornl.gov kernel: [701665.600780] Lustre: atlas2-OST0097: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8807fbf5c000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4a1.ccs.ornl.gov kernel: [1915354.845420] Lustre: atlas2-OST00d8: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8808395e1000, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4a1.ccs.ornl.gov kernel: [1915354.916812] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss3d5.ccs.ornl.gov kernel: [1915360.379171] Lustre: atlas2-OST00ac: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806f5dbb000, cur 1405630601 expire 1405629701 last 1405629247 Jul 17 16:56:41 atlas-oss4i1.ccs.ornl.gov kernel: [1915357.017965] Lustre: atlas2-OST0238: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880638942800, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4i1.ccs.ornl.gov kernel: [1915357.086988] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss4h7.ccs.ornl.gov kernel: [1915365.176988] Lustre: atlas2-OST0236: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8801ec9a1400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4h7.ccs.ornl.gov kernel: [1915365.248085] Lustre: Skipped 1 previous similar message Jul 17 16:56:41 atlas-oss3f6.ccs.ornl.gov kernel: [802061.474305] Lustre: atlas2-OST00bd: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880872d19400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3d3.ccs.ornl.gov kernel: [1915363.719612] Lustre: atlas2-OST00aa: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806ee28e400, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss3h7.ccs.ornl.gov kernel: [1915363.605062] Lustre: atlas2-OST003e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff880572b30c00, cur 1405630601 expire 1405629701 last 1405629248 Jul 17 16:56:41 atlas-oss4f2.ccs.ornl.gov kernel: [1915360.519637] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss4c1.ccs.ornl.gov kernel: [1915362.904258] Lustre: atlas2-OST0298: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88056f2bd000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4b2.ccs.ornl.gov kernel: [1915380.210390] Lustre: atlas2-OST0291: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88036b875000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4c1.ccs.ornl.gov kernel: [1915362.967763] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4b2.ccs.ornl.gov kernel: [1915380.274215] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3g5.ccs.ornl.gov kernel: [1913958.993329] Lustre: atlas2-OST0304: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8807803c2000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3g5.ccs.ornl.gov kernel: [1913959.057517] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3g8.ccs.ornl.gov kernel: [1913955.456814] Lustre: atlas2-OST0307: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff8806f0aae000, cur 1405630602 expire 1405629702 last 1405629249 Jul 17 16:56:42 atlas-oss3g4.ccs.ornl.gov kernel: [1913967.908311] Lustre: atlas2-OST0273: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8805b757c800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3g4.ccs.ornl.gov kernel: [1913967.978658] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss3g7.ccs.ornl.gov kernel: [1913954.967240] Lustre: atlas2-OST0276: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff88050fdaf000, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss3g3.ccs.ornl.gov kernel: [1913954.799639] Lustre: atlas2-OST0302: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8805a3eed800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3g7.ccs.ornl.gov kernel: [1913955.037910] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss3g3.ccs.ornl.gov kernel: [1913954.869638] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3h4.ccs.ornl.gov kernel: [1915361.187748] Lustre: atlas2-OST027b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff8801b9041400, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss3g8.ccs.ornl.gov kernel: [1913955.524125] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4b4.ccs.ornl.gov kernel: [1915355.524134] Lustre: atlas2-OST0293: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8808704b3400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4b4.ccs.ornl.gov kernel: [1915355.603536] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3h4.ccs.ornl.gov kernel: [1915361.259570] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss4g4.ccs.ornl.gov kernel: [1915372.383825] Lustre: atlas2-OST034b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff880839540c00, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss4g4.ccs.ornl.gov kernel: [1915372.438718] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss3c8.ccs.ornl.gov kernel: [1915362.987888] Lustre: atlas2-OST02e7: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8804d9f53c00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3c8.ccs.ornl.gov kernel: [1915363.049938] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3d2.ccs.ornl.gov kernel: [1915359.653695] Lustre: atlas2-OST02e9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88044678fc00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3d2.ccs.ornl.gov kernel: [1915359.717047] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3e1.ccs.ornl.gov kernel: [1915360.685227] Lustre: atlas2-OST02f0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8807995ffc00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3e1.ccs.ornl.gov kernel: [1915360.748685] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4d6.ccs.ornl.gov kernel: [1915358.027744] Lustre: atlas2-OST0335: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8804231ba800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4d6.ccs.ornl.gov kernel: [1915358.091103] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4f8.ccs.ornl.gov kernel: [1915357.193132] Lustre: atlas2-OST0347: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880870c2a800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4d2.ccs.ornl.gov kernel: [1915363.033847] Lustre: atlas2-OST02a1: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88072fec1c00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4c8.ccs.ornl.gov kernel: [1915349.800486] Lustre: atlas2-OST03bf: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8804d4e20000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4c8.ccs.ornl.gov kernel: [1915349.863805] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4e5.ccs.ornl.gov kernel: [1915359.026353] Lustre: atlas2-OST03cc: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff8803e671d000, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss4e5.ccs.ornl.gov kernel: [1915359.089398] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3h3.ccs.ornl.gov kernel: [1915353.531531] Lustre: atlas2-OST030a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8804b15c4400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4b1.ccs.ornl.gov kernel: [1915362.430990] Lustre: atlas2-OST03b0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880527cd5000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4b1.ccs.ornl.gov kernel: [1915362.494206] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss3h3.ccs.ornl.gov kernel: [1915353.595141] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3e2.ccs.ornl.gov kernel: [1915357.332914] Lustre: atlas2-OST0381: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8801155f3800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4h4.ccs.ornl.gov kernel: [1915377.343013] Lustre: atlas2-OST0353: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88056c0e9400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4f5.ccs.ornl.gov kernel: [1915355.860816] Lustre: atlas2-OST03d4: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1353 seconds. I think it's dead, and I am evicting it. exp ffff88045c371800, cur 1405630602 expire 1405629702 last 1405629249 Jul 17 16:56:42 atlas-oss3f8.ccs.ornl.gov kernel: [802060.336577] Lustre: atlas2-OST038f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff88083c41d800, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss4b5.ccs.ornl.gov kernel: [1915375.499845] Lustre: atlas2-OST03b4: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8808713f1000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3e2.ccs.ornl.gov kernel: [1915357.396080] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4h4.ccs.ornl.gov kernel: [1915377.406119] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss4i6.ccs.ornl.gov kernel: [1915371.319587] Lustre: atlas2-OST03ed: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88056c70b000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3f4.ccs.ornl.gov kernel: [1915360.109148] Lustre: atlas2-OST038b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88081cd5d400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4i4.ccs.ornl.gov kernel: [1915362.841310] Lustre: atlas2-OST035b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880824edcc00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3f8.ccs.ornl.gov kernel: [802060.400363] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3b5.ccs.ornl.gov kernel: [1915370.945701] Lustre: atlas2-OST036c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806f665d800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4d7.ccs.ornl.gov kernel: [1915358.635274] Lustre: atlas2-OST03c6: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff8803b568b800, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss4a7.ccs.ornl.gov kernel: [1915369.598847] Lustre: atlas2-OST03ae: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880755ca9c00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3d3.ccs.ornl.gov kernel: [1915364.765881] Lustre: atlas2-OST037a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8808713ea800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4h5.ccs.ornl.gov kernel: [1915360.441983] Lustre: atlas2-OST0354: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff880689d8b000, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss3f7.ccs.ornl.gov kernel: [801689.000478] Lustre: atlas2-OST038e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff880818b3cc00, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss4c2.ccs.ornl.gov kernel: [1915361.869350] Lustre: atlas2-OST0329: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880766efec00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3f4.ccs.ornl.gov kernel: [1915360.172703] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4d8.ccs.ornl.gov kernel: [1915370.991249] Lustre: atlas2-OST03c7: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806beb27400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4i4.ccs.ornl.gov kernel: [1915362.904757] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss4i6.ccs.ornl.gov kernel: [1915371.383358] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3b5.ccs.ornl.gov kernel: [1915371.009156] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4d7.ccs.ornl.gov kernel: [1915358.699304] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3a2.ccs.ornl.gov kernel: [1915357.472730] Lustre: atlas2-OST02d1: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88086f272000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4a7.ccs.ornl.gov kernel: [1915369.662564] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4i5.ccs.ornl.gov kernel: [1915365.891351] Lustre: atlas2-OST035c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880862fc8000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3f7.ccs.ornl.gov kernel: [801689.063997] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4h5.ccs.ornl.gov kernel: [1915360.506330] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4d8.ccs.ornl.gov kernel: [1915371.055249] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4c2.ccs.ornl.gov kernel: [1915361.932952] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3a2.ccs.ornl.gov kernel: [1915357.536498] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3h1.ccs.ornl.gov kernel: [1915346.650433] Lustre: atlas2-OST0398: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8805af027000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3d6.ccs.ornl.gov kernel: [1915372.323427] Lustre: atlas2-OST037d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806bb552c00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3i4.ccs.ornl.gov kernel: [1915363.806764] Lustre: atlas2-OST03a3: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff8805402eec00, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss3e5.ccs.ornl.gov kernel: [1915359.185934] Lustre: atlas2-OST02f4: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806dec3c800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3i4.ccs.ornl.gov kernel: [1915363.871446] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3c3.ccs.ornl.gov kernel: [1915350.632221] Lustre: atlas2-OST0372: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880124793400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3c3.ccs.ornl.gov kernel: [1915350.697087] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3e5.ccs.ornl.gov kernel: [1915359.250973] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4e1.ccs.ornl.gov kernel: [1915355.802894] Lustre: atlas2-OST03c8: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806abc77400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4e1.ccs.ornl.gov kernel: [1915355.867520] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4f6.ccs.ornl.gov kernel: [1915352.711957] Lustre: atlas2-OST0345: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff8808390aa800, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss3b4.ccs.ornl.gov kernel: [1915369.513070] Lustre: atlas2-OST036b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88036322f000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4f6.ccs.ornl.gov kernel: [1915352.776580] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss3i6.ccs.ornl.gov kernel: [1915356.800972] Lustre: atlas2-OST0315: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88021ab41800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4c3.ccs.ornl.gov kernel: [1915354.721137] Lustre: atlas2-OST03ba: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8802a7c83400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3i1.ccs.ornl.gov kernel: [1915358.088131] Lustre: atlas2-OST03a0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880489894800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3i6.ccs.ornl.gov kernel: [1915356.866652] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3f2.ccs.ornl.gov kernel: [1915353.076898] Lustre: atlas2-OST02f9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88073ae22800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3h6.ccs.ornl.gov kernel: [1915358.317313] Lustre: atlas2-OST039d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880649fb9400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4c3.ccs.ornl.gov kernel: [1915354.784487] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4i2.ccs.ornl.gov kernel: [1915380.800174] Lustre: atlas2-OST03e9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806c681ec00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4d1.ccs.ornl.gov kernel: [1915353.970925] Lustre: atlas2-OST03c0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88078e532000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4g3.ccs.ornl.gov kernel: [1915369.081319] Lustre: atlas2-OST03da: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806fcabc000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3d7.ccs.ornl.gov kernel: [1915358.812089] Lustre: atlas2-OST02ee: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff8806ed057c00, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss4a5.ccs.ornl.gov kernel: [1915376.895714] Lustre: atlas2-OST03ac: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88086ba59800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4b6.ccs.ornl.gov kernel: [1915355.555787] Lustre: atlas2-OST03b5: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880870360400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3d1.ccs.ornl.gov kernel: [1915350.610881] Lustre: atlas2-OST02e8: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880716ca9000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3h6.ccs.ornl.gov kernel: [1915358.383006] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3f2.ccs.ornl.gov kernel: [1915353.142799] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4i2.ccs.ornl.gov kernel: [1915380.863791] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4e8.ccs.ornl.gov kernel: [1915370.894730] Lustre: atlas2-OST033f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880278f58000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4g2.ccs.ornl.gov kernel: [1915368.705497] Lustre: atlas2-OST03d9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff880282f89400, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss3e7.ccs.ornl.gov kernel: [1915360.334470] Lustre: atlas2-OST0386: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8808708b2000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3i5.ccs.ornl.gov kernel: [701665.762703] Lustre: atlas2-OST03a4: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880fff911400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4g3.ccs.ornl.gov kernel: [1915369.144870] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4a5.ccs.ornl.gov kernel: [1915376.961563] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3d1.ccs.ornl.gov kernel: [1915350.677088] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss4e4.ccs.ornl.gov kernel: [1915366.629475] Lustre: atlas2-OST02ab: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806bf328000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3g1.ccs.ornl.gov kernel: [1913947.904876] Lustre: atlas2-OST0390: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8805da8de000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4g2.ccs.ornl.gov kernel: [1915368.769177] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4e8.ccs.ornl.gov kernel: [1915370.960835] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3a8.ccs.ornl.gov kernel: [701666.647375] Lustre: atlas2-OST0367: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff881015976400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3i5.ccs.ornl.gov kernel: [701665.828241] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3e7.ccs.ornl.gov kernel: [1915360.399921] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4h1.ccs.ornl.gov kernel: [1915351.821847] Lustre: atlas2-OST02c0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88050d4b9400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3g1.ccs.ornl.gov kernel: [1913947.970472] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3a8.ccs.ornl.gov kernel: [701666.713041] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4e2.ccs.ornl.gov kernel: [1915371.742905] Lustre: atlas2-OST03c9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff880808454c00, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss4h1.ccs.ornl.gov kernel: [1915351.888540] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss4f2.ccs.ornl.gov kernel: [1915361.504680] Lustre: atlas2-OST03d1: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8802419de000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4a1.ccs.ornl.gov kernel: [1915355.890412] Lustre: atlas2-OST0318: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff8807bd6e8400, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss4e2.ccs.ornl.gov kernel: [1915371.807121] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4d5.ccs.ornl.gov kernel: [1915370.447885] Lustre: atlas2-OST0334: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880362b3d000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4i3.ccs.ornl.gov kernel: [1915350.172707] Lustre: atlas2-OST035a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8804e368a400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4a1.ccs.ornl.gov kernel: [1915355.957219] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss3h5.ccs.ornl.gov kernel: [1915377.642400] Lustre: atlas2-OST039c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880269394000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4d5.ccs.ornl.gov kernel: [1915370.513864] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss4i3.ccs.ornl.gov kernel: [1915350.238826] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss4c7.ccs.ornl.gov kernel: [1915357.810331] Lustre: atlas2-OST032e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8807721ce800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3h5.ccs.ornl.gov kernel: [1915377.709008] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4f7.ccs.ornl.gov kernel: [1915357.004561] Lustre: atlas2-OST0346: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880825b44000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4c7.ccs.ornl.gov kernel: [1915357.877484] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4a3.ccs.ornl.gov kernel: [1915373.901009] Lustre: atlas2-OST03aa: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff88086fd7d400, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss4f7.ccs.ornl.gov kernel: [1915357.071755] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss4g7.ccs.ornl.gov kernel: [1915380.235675] Lustre: atlas2-OST03de: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806b481d800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4e7.ccs.ornl.gov kernel: [1915355.348884] Lustre: atlas2-OST03ce: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8802ce054000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4g7.ccs.ornl.gov kernel: [1915380.300804] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3a3.ccs.ornl.gov kernel: [1915355.179243] Lustre: atlas2-OST0362: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880870b1b400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3f5.ccs.ornl.gov kernel: [802049.838042] Lustre: atlas2-OST038c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880871b70800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4b8.ccs.ornl.gov kernel: [1915359.794301] Lustre: atlas2-OST0327: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880447c20800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4i1.ccs.ornl.gov kernel: [1915358.061961] Lustre: atlas2-OST03e8: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880526c47400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4g8.ccs.ornl.gov kernel: [1915357.028620] Lustre: atlas2-OST034f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880657be9c00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4i8.ccs.ornl.gov kernel: [1915371.634249] Lustre: atlas2-OST02cf: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880312afb000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3f1.ccs.ornl.gov kernel: [1915361.777229] Lustre: atlas2-OST0388: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8801f604c000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3a3.ccs.ornl.gov kernel: [1915355.246263] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3b6.ccs.ornl.gov kernel: [1915356.172145] Lustre: atlas2-OST036d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880870503800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3f5.ccs.ornl.gov kernel: [802049.905816] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4b8.ccs.ornl.gov kernel: [1915359.862241] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss3b2.ccs.ornl.gov kernel: [1915357.939199] Lustre: atlas2-OST0369: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88086ed18c00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4g8.ccs.ornl.gov kernel: [1915357.095740] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4i1.ccs.ornl.gov kernel: [1915358.127587] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3b6.ccs.ornl.gov kernel: [1915356.240081] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3f1.ccs.ornl.gov kernel: [1915361.844793] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4h2.ccs.ornl.gov kernel: [1915379.159210] Lustre: atlas2-OST0351: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880871244c00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3g6.ccs.ornl.gov kernel: [1913966.312528] Lustre: atlas2-OST0395: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff88058ee38800, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss3f3.ccs.ornl.gov kernel: [1915371.469958] Lustre: atlas2-OST038a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88052412d000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3a5.ccs.ornl.gov kernel: [1915351.468867] Lustre: atlas2-OST0364: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880259d2ac00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3c4.ccs.ornl.gov kernel: [1915353.231349] Lustre: atlas2-OST0373: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1357 seconds. I think it's dead, and I am evicting it. exp ffff88013b08d800, cur 1405630602 expire 1405629702 last 1405629245 Jul 17 16:56:42 atlas-oss3e4.ccs.ornl.gov kernel: [1915372.183568] Lustre: atlas2-OST02f3: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88045935d000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3e6.ccs.ornl.gov kernel: [1915355.512127] Lustre: atlas2-OST0385: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806f4963400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3c6.ccs.ornl.gov kernel: [1915350.196513] Lustre: atlas2-OST0375: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88083990dc00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3g6.ccs.ornl.gov kernel: [1913966.380462] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4c5.ccs.ornl.gov kernel: [1915353.684909] Lustre: atlas2-OST029c: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8808718ccc00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3d4.ccs.ornl.gov kernel: [701666.087779] Lustre: atlas2-OST037b: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88100a59a800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4h8.ccs.ornl.gov kernel: [1915348.879020] Lustre: atlas2-OST0357: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff88055835b000, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss3f3.ccs.ornl.gov kernel: [1915371.538022] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4f4.ccs.ornl.gov kernel: [1915358.398351] Lustre: atlas2-OST0343: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8805ab11c800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3e4.ccs.ornl.gov kernel: [1915372.252371] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3c4.ccs.ornl.gov kernel: [1915353.299406] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3c6.ccs.ornl.gov kernel: [1915350.265282] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3d4.ccs.ornl.gov kernel: [701666.151348] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3e6.ccs.ornl.gov kernel: [1915355.580417] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4f4.ccs.ornl.gov kernel: [1915358.467224] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss4h8.ccs.ornl.gov kernel: [1915348.947130] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4c5.ccs.ornl.gov kernel: [1915353.754254] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss3h2.ccs.ornl.gov kernel: [1915360.980971] Lustre: atlas2-OST0399: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1357 seconds. I think it's dead, and I am evicting it. exp ffff8801d4c84800, cur 1405630602 expire 1405629702 last 1405629245 Jul 17 16:56:42 atlas-oss3b8.ccs.ornl.gov kernel: [1915372.094892] Lustre: atlas2-OST036f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880737146400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3h2.ccs.ornl.gov kernel: [1915361.049570] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3b8.ccs.ornl.gov kernel: [1915372.163363] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3b7.ccs.ornl.gov kernel: [1915362.830105] Lustre: atlas2-OST036e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88086ec3b400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4c6.ccs.ornl.gov kernel: [1915358.853565] Lustre: atlas2-OST03bd: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8805e0310000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4g6.ccs.ornl.gov kernel: [1915367.613019] Lustre: atlas2-OST034d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88075ce8f400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4c6.ccs.ornl.gov kernel: [1915358.922481] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4a2.ccs.ornl.gov kernel: [1915372.565263] Lustre: atlas2-OST03a9: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880444d38800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3b7.ccs.ornl.gov kernel: [1915362.899439] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4b7.ccs.ornl.gov kernel: [1915352.000205] Lustre: atlas2-OST0296: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8808709c2400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4g6.ccs.ornl.gov kernel: [1915367.682467] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4a2.ccs.ornl.gov kernel: [1915372.634332] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4h7.ccs.ornl.gov kernel: [1915366.218778] Lustre: atlas2-OST0356: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff880752b55400, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss4b7.ccs.ornl.gov kernel: [1915352.070220] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3c7.ccs.ornl.gov kernel: [1915358.815520] Lustre: atlas2-OST0376: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88070fcc4c00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4h7.ccs.ornl.gov kernel: [1915366.288631] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss3c7.ccs.ornl.gov kernel: [1915358.884862] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4a6.ccs.ornl.gov kernel: [1915360.007856] Lustre: atlas2-OST03ad: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880737672800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3i3.ccs.ornl.gov kernel: [1915353.414294] Lustre: atlas2-OST0312: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1356 seconds. I think it's dead, and I am evicting it. exp ffff88087351bc00, cur 1405630602 expire 1405629702 last 1405629246 Jul 17 16:56:42 atlas-oss3e8.ccs.ornl.gov kernel: [1915358.061446] Lustre: atlas2-OST02f7: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1356 seconds. I think it's dead, and I am evicting it. exp ffff8805509d0c00, cur 1405630602 expire 1405629702 last 1405629246 Jul 17 16:56:42 atlas-oss4b3.ccs.ornl.gov kernel: [1915352.241501] Lustre: atlas2-OST03b2: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88081df76c00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4f3.ccs.ornl.gov kernel: [1915352.666996] Lustre: atlas2-OST02b2: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8805f8046800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4a6.ccs.ornl.gov kernel: [1915360.077543] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss3i3.ccs.ornl.gov kernel: [1915353.484717] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss3c1.ccs.ornl.gov kernel: [1915360.066872] Lustre: atlas2-OST0370: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff8806e1527c00, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss4a8.ccs.ornl.gov kernel: [1915363.988533] Lustre: atlas2-OST03af: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff88073daf7c00, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss3h8.ccs.ornl.gov kernel: [1915354.805502] Lustre: atlas2-OST030f: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880342df2000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3g2.ccs.ornl.gov kernel: [1913954.574740] Lustre: atlas2-OST0391: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8805db03e400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3e8.ccs.ornl.gov kernel: [1915358.131992] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3c1.ccs.ornl.gov kernel: [1915360.136673] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4b3.ccs.ornl.gov kernel: [1915352.311639] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4a8.ccs.ornl.gov kernel: [1915364.056791] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3a6.ccs.ornl.gov kernel: [1915360.566105] Lustre: atlas2-OST02d5: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88087002e800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4f3.ccs.ornl.gov kernel: [1915352.736608] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4g5.ccs.ornl.gov kernel: [1915358.047110] Lustre: atlas2-OST03dc: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806b53bc800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3h8.ccs.ornl.gov kernel: [1915354.876160] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3g2.ccs.ornl.gov kernel: [1913954.645226] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3i7.ccs.ornl.gov kernel: [1915355.087344] Lustre: atlas2-OST0316: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88086fc15000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3a6.ccs.ornl.gov kernel: [1915360.637013] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4d3.ccs.ornl.gov kernel: [1915354.408489] Lustre: atlas2-OST0332: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88042279d400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4g5.ccs.ornl.gov kernel: [1915358.117504] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3i7.ccs.ornl.gov kernel: [1915355.157518] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4g1.ccs.ornl.gov kernel: [1915355.022358] Lustre: atlas2-OST0348: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8806f4db3800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4d3.ccs.ornl.gov kernel: [1915354.478764] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss4g1.ccs.ornl.gov kernel: [1915355.092817] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss4i7.ccs.ornl.gov kernel: [1915371.215671] Lustre: atlas2-OST035e: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88026c13dc00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4i7.ccs.ornl.gov kernel: [1915371.286353] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3b3.ccs.ornl.gov kernel: [1915350.909512] Lustre: atlas2-OST036a: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880475267000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4c4.ccs.ornl.gov kernel: [1915369.800114] Lustre: atlas2-OST03bb: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff88014bbde400, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3c2.ccs.ornl.gov kernel: [1915359.993106] Lustre: atlas2-OST0371: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff8803fe6c8c00, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss4c4.ccs.ornl.gov kernel: [1915369.871742] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3c2.ccs.ornl.gov kernel: [1915360.064592] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss4d4.ccs.ornl.gov kernel: [1915378.282354] Lustre: atlas2-OST03c3: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff880871b69000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss3f6.ccs.ornl.gov kernel: [802062.520350] Lustre: atlas2-OST038d: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1357 seconds. I think it's dead, and I am evicting it. exp ffff88085f27f400, cur 1405630602 expire 1405629702 last 1405629245 Jul 17 16:56:42 atlas-oss4d4.ccs.ornl.gov kernel: [1915378.345742] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4h3.ccs.ornl.gov kernel: [1915370.629778] Lustre: atlas2-OST03e2: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff88055b01a400, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss3i8.ccs.ornl.gov kernel: [1915371.832721] Lustre: atlas2-OST03a7: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff880805c63800, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss3i8.ccs.ornl.gov kernel: [1915371.906617] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4h3.ccs.ornl.gov kernel: [1915370.701463] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4e3.ccs.ornl.gov kernel: [1915363.754601] Lustre: atlas2-OST03ca: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8805e56d5c00, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4e3.ccs.ornl.gov kernel: [1915363.826517] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4f1.ccs.ornl.gov kernel: [1915362.113894] Lustre: atlas2-OST02b0: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff88047fe0f000, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss4h6.ccs.ornl.gov kernel: [1915356.737033] Lustre: atlas2-OST03e5: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8808391c9800, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4a4.ccs.ornl.gov kernel: [1915359.488936] Lustre: atlas2-OST03ab: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1354 seconds. I think it's dead, and I am evicting it. exp ffff8804404fc000, cur 1405630602 expire 1405629702 last 1405629248 Jul 17 16:56:42 atlas-oss4f1.ccs.ornl.gov kernel: [1915362.187571] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4a4.ccs.ornl.gov kernel: [1915359.563361] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3h1.ccs.ornl.gov kernel: [1915346.714657] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3d6.ccs.ornl.gov kernel: [1915372.388242] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3i1.ccs.ornl.gov kernel: [1915358.153420] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3b4.ccs.ornl.gov kernel: [1915369.578037] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3d7.ccs.ornl.gov kernel: [1915358.877367] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4e4.ccs.ornl.gov kernel: [1915366.696100] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4a3.ccs.ornl.gov kernel: [1915373.968098] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4e7.ccs.ornl.gov kernel: [1915355.414225] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4i8.ccs.ornl.gov kernel: [1915371.701513] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3b2.ccs.ornl.gov kernel: [1915358.006753] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4e6.ccs.ornl.gov kernel: [1915373.691764] Lustre: atlas2-OST02ad: haven't heard from client atlas2-MDT0000-mdtlov_UUID (at 10.36.226.74@o2ib) in 1355 seconds. I think it's dead, and I am evicting it. exp ffff8802700ed800, cur 1405630602 expire 1405629702 last 1405629247 Jul 17 16:56:42 atlas-oss4e6.ccs.ornl.gov kernel: [1915373.771688] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss3a5.ccs.ornl.gov kernel: [1915351.536881] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4d2.ccs.ornl.gov kernel: [1915363.097446] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss4f8.ccs.ornl.gov kernel: [1915357.256549] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4b5.ccs.ornl.gov kernel: [1915375.563440] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4f5.ccs.ornl.gov kernel: [1915355.924365] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3f6.ccs.ornl.gov kernel: [802062.592253] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss3d3.ccs.ornl.gov kernel: [1915364.829683] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4i5.ccs.ornl.gov kernel: [1915365.955179] Lustre: Skipped 1 previous similar message Jul 17 16:56:42 atlas-oss4d1.ccs.ornl.gov kernel: [1915354.036643] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4h6.ccs.ornl.gov kernel: [1915356.809317] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4b6.ccs.ornl.gov kernel: [1915355.621263] Lustre: Skipped 3 previous similar messages Jul 17 16:56:42 atlas-oss4f2.ccs.ornl.gov kernel: [1915361.569175] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss4h2.ccs.ornl.gov kernel: [1915379.226990] Lustre: Skipped 2 previous similar messages Jul 17 16:56:42 atlas-oss3b3.ccs.ornl.gov kernel: [1915350.981386] Lustre: Skipped 2 previous similar messages Jul 17 16:59:37 atlas-oss4g5.ccs.ornl.gov kernel: [1915532.229179] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630496/real 1405630496] req@ffff8804f86a5400 x1471904510456824/t0(0) o38->atlas2-MDT0000-lwp-OST034c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630776 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4g5.ccs.ornl.gov kernel: [1915532.322180] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4g2.ccs.ornl.gov kernel: [1915542.900303] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630496/real 1405630496] req@ffff8808708a1400 x1471904509229056/t0(0) o38->atlas2-MDT0000-lwp-OST03d9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630776 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4g2.ccs.ornl.gov kernel: [1915542.985974] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4e4.ccs.ornl.gov kernel: [1915540.817834] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630496/real 1405630496] req@ffff8803684ea400 x1471904515374396/t0(0) o38->atlas2-MDT0000-lwp-OST021b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630776 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4e4.ccs.ornl.gov kernel: [1915540.912902] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4h7.ccs.ornl.gov kernel: [1915540.421717] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630496/real 1405630496] req@ffff8806f2510800 x1471904512615220/t0(0) o38->atlas2-MDT0000-lwp-OST0236@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630776 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4h7.ccs.ornl.gov kernel: [1915540.514782] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4d2.ccs.ornl.gov kernel: [1915537.241515] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630496/real 1405630496] req@ffff8805b722d800 x1471904512221044/t0(0) o38->atlas2-MDT0000-lwp-OST03c1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630776 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4d2.ccs.ornl.gov kernel: [1915537.333273] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4h6.ccs.ornl.gov kernel: [1915530.948596] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630496/real 1405630496] req@ffff880fd2b8dc00 x1471904515532184/t0(0) o38->atlas2-MDT0000-lwp-OST03e5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630776 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4h6.ccs.ornl.gov kernel: [1915531.035301] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4i2.ccs.ornl.gov kernel: [1915555.016809] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630496/real 1405630496] req@ffff8801ffaff400 x1471904514634616/t0(0) o38->atlas2-MDT0000-lwp-OST03e9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630776 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4i2.ccs.ornl.gov kernel: [1915555.101809] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4i7.ccs.ornl.gov kernel: [1915545.429996] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630496/real 1405630496] req@ffff880fc5204000 x1471904511375192/t0(0) o38->atlas2-MDT0000-lwp-OST035e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630776 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4b3.ccs.ornl.gov kernel: [1915526.460522] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630496/real 1405630496] req@ffff88037035d400 x1471904509180604/t0(0) o38->atlas2-MDT0000-lwp-OST03b2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630776 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4i7.ccs.ornl.gov kernel: [1915545.523040] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4f7.ccs.ornl.gov kernel: [1915531.230763] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630496/real 1405630496] req@ffff8805e0df4800 x1471904510455128/t0(0) o38->atlas2-MDT0000-lwp-OST0346@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630776 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4f7.ccs.ornl.gov kernel: [1915531.317860] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4h2.ccs.ornl.gov kernel: [1915553.391706] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630496/real 1405630496] req@ffff8801035d3c00 x1471904508411220/t0(0) o38->atlas2-MDT0000-lwp-OST03e1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630776 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4b3.ccs.ornl.gov kernel: [1915526.547452] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4b2.ccs.ornl.gov kernel: [1915554.989020] Lustre: 27461:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880f45e4a400 x1471904518025928/t0(0) o38->atlas2-MDT0000-lwp-OST03b1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4b2.ccs.ornl.gov kernel: [1915555.081737] Lustre: 27461:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 16:59:37 atlas-oss3e3.ccs.ornl.gov kernel: [1915528.886217] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88015a9dd800 x1471904510468524/t0(0) o38->atlas2-MDT0000-lwp-OST0262@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3e3.ccs.ornl.gov kernel: [1915528.977772] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4h2.ccs.ornl.gov kernel: [1915553.484164] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4i3.ccs.ornl.gov kernel: [1915524.448617] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880d292f8000 x1471904523159664/t0(0) o38->atlas2-MDT0000-lwp-OST03ea@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4i3.ccs.ornl.gov kernel: [1915524.533891] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4d7.ccs.ornl.gov kernel: [1915532.919470] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88104bf2dc00 x1471904514720064/t0(0) o38->atlas2-MDT0000-lwp-OST03c6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4d7.ccs.ornl.gov kernel: [1915533.005860] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3h5.ccs.ornl.gov kernel: [1915551.931844] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8806b9bea000 x1471904507806668/t0(0) o38->atlas2-MDT0000-lwp-OST039c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3h5.ccs.ornl.gov kernel: [1915552.016834] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3f6.ccs.ornl.gov kernel: [802236.815140] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880d1aafbc00 x1473076856558664/t0(0) o38->atlas2-MDT0000-lwp-OST038d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3f6.ccs.ornl.gov kernel: [802236.899463] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4i4.ccs.ornl.gov kernel: [1915537.134908] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880ffa0b3400 x1471904516513512/t0(0) o38->atlas2-MDT0000-lwp-OST03eb@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4i4.ccs.ornl.gov kernel: [1915537.221317] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3c1.ccs.ornl.gov kernel: [1915534.367436] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8804da4c4400 x1471904511393040/t0(0) o38->atlas2-MDT0000-lwp-OST01c0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3i7.ccs.ornl.gov kernel: [1915529.388057] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880fee6c0800 x1471904508389376/t0(0) o38->atlas2-MDT0000-lwp-OST03a6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4f2.ccs.ornl.gov kernel: [1915535.811806] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff881050d83c00 x1471904508910764/t0(0) o38->atlas2-MDT0000-lwp-OST03d1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4d3.ccs.ornl.gov kernel: [1915528.719889] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff881029ac1000 x1471904516978344/t0(0) o38->atlas2-MDT0000-lwp-OST03c2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4d3.ccs.ornl.gov kernel: [1915528.804970] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4c5.ccs.ornl.gov kernel: [1915528.002851] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88086f2e1800 x1471904510171304/t0(0) o38->atlas2-MDT0000-lwp-OST032c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4c4.ccs.ornl.gov kernel: [1915544.114484] Lustre: 25786:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8810110f2000 x1471904510981140/t0(0) o38->atlas2-MDT0000-lwp-OST03bb@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4c4.ccs.ornl.gov kernel: [1915544.207907] Lustre: 25786:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4h4.ccs.ornl.gov kernel: [1915551.662851] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88036a95f800 x1471904512334440/t0(0) o38->atlas2-MDT0000-lwp-OST03e3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4h4.ccs.ornl.gov kernel: [1915551.752903] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4c3.ccs.ornl.gov kernel: [1915529.043140] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8808bd0ef800 x1471904512246156/t0(0) o38->atlas2-MDT0000-lwp-OST032a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4c3.ccs.ornl.gov kernel: [1915529.130504] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3h8.ccs.ornl.gov kernel: [1915529.137512] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8804aea3e800 x1471904514772900/t0(0) o38->atlas2-MDT0000-lwp-OST039f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3h8.ccs.ornl.gov kernel: [1915529.222101] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3c1.ccs.ornl.gov kernel: [1915534.453271] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3i7.ccs.ornl.gov kernel: [1915529.473736] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3f4.ccs.ornl.gov kernel: [1915534.443495] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880d04428c00 x1471904507778548/t0(0) o38->atlas2-MDT0000-lwp-OST026b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3f4.ccs.ornl.gov kernel: [1915534.528329] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3d6.ccs.ornl.gov kernel: [1915546.658332] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880cd7f40000 x1471904511183460/t0(0) o38->atlas2-MDT0000-lwp-OST037d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3d6.ccs.ornl.gov kernel: [1915546.746387] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3g8.ccs.ornl.gov kernel: [1914130.317481] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8801e651b000 x1471904513602496/t0(0) o38->atlas2-MDT0000-lwp-OST0397@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3g8.ccs.ornl.gov kernel: [1914130.410440] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4a4.ccs.ornl.gov kernel: [1915533.826633] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88037d78f400 x1471904509591176/t0(0) o38->atlas2-MDT0000-lwp-OST03ab@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4a4.ccs.ornl.gov kernel: [1915533.919509] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4f2.ccs.ornl.gov kernel: [1915535.896425] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4a5.ccs.ornl.gov kernel: [1915551.245561] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880a0f5a3800 x1471904519454224/t0(0) o38->atlas2-MDT0000-lwp-OST03ac@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4a5.ccs.ornl.gov kernel: [1915551.330031] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3e1.ccs.ornl.gov kernel: [1915535.039787] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88078c5ab000 x1471904509694424/t0(0) o38->atlas2-MDT0000-lwp-OST02f0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3e1.ccs.ornl.gov kernel: [1915535.124394] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4c5.ccs.ornl.gov kernel: [1915528.090445] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4a2.ccs.ornl.gov kernel: [1915546.924798] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8803122e8c00 x1471904510895776/t0(0) o38->atlas2-MDT0000-lwp-OST01f9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4c8.ccs.ornl.gov kernel: [1915524.156542] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88094d777400 x1471904512196984/t0(0) o38->atlas2-MDT0000-lwp-OST020f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3f8.ccs.ornl.gov kernel: [802234.700204] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880ecdafcc00 x1473076861321020/t0(0) o38->atlas2-MDT0000-lwp-OST038f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3f8.ccs.ornl.gov kernel: [802234.786667] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3f5.ccs.ornl.gov kernel: [802224.197769] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88092781a000 x1473076856406372/t0(0) o38->atlas2-MDT0000-lwp-OST038c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3f5.ccs.ornl.gov kernel: [802224.290727] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4e7.ccs.ornl.gov kernel: [1915529.706880] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88086bfc3400 x1471904522621592/t0(0) o38->atlas2-MDT0000-lwp-OST03ce@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4e7.ccs.ornl.gov kernel: [1915529.799864] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3g5.ccs.ornl.gov kernel: [1914133.898319] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880219fda000 x1471904518249156/t0(0) o38->atlas2-MDT0000-lwp-OST0304@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3g5.ccs.ornl.gov kernel: [1914133.983844] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4c2.ccs.ornl.gov kernel: [1915536.246692] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8808a4da8800 x1471904508570740/t0(0) o38->atlas2-MDT0000-lwp-OST0329@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4c2.ccs.ornl.gov kernel: [1915536.339060] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4a2.ccs.ornl.gov kernel: [1915547.011820] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4c8.ccs.ornl.gov kernel: [1915524.248336] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4e1.ccs.ornl.gov kernel: [1915530.199733] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880fa5767000 x1471904515144832/t0(0) o38->atlas2-MDT0000-lwp-OST03c8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4e1.ccs.ornl.gov kernel: [1915530.292692] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4d4.ccs.ornl.gov kernel: [1915552.688981] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8801db95e400 x1471904513679232/t0(0) o38->atlas2-MDT0000-lwp-OST03c3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4d4.ccs.ornl.gov kernel: [1915552.773473] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3h3.ccs.ornl.gov kernel: [1915527.943238] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8806773efc00 x1471904512413684/t0(0) o38->atlas2-MDT0000-lwp-OST039a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3h3.ccs.ornl.gov kernel: [1915528.030431] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3h7.ccs.ornl.gov kernel: [1915539.067245] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8807b1d13800 x1471904510333020/t0(0) o38->atlas2-MDT0000-lwp-OST039e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3h7.ccs.ornl.gov kernel: [1915539.153358] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3a1.ccs.ornl.gov kernel: [1915527.175585] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880ddedabc00 x1471904525931840/t0(0) o38->atlas2-MDT0000-lwp-OST0360@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3a1.ccs.ornl.gov kernel: [1915527.262874] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4e2.ccs.ornl.gov kernel: [1915546.159533] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880dd51f2400 x1471904514146420/t0(0) o38->atlas2-MDT0000-lwp-OST03c9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4e2.ccs.ornl.gov kernel: [1915546.244152] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4g8.ccs.ornl.gov kernel: [1915531.440750] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880483424800 x1471904521746900/t0(0) o38->atlas2-MDT0000-lwp-OST034f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4h8.ccs.ornl.gov kernel: [1915523.298248] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880360b7e800 x1471904514267280/t0(0) o38->atlas2-MDT0000-lwp-OST03e7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4h8.ccs.ornl.gov kernel: [1915523.383180] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3f7.ccs.ornl.gov kernel: [801863.416425] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88074e983400 x1473076863306268/t0(0) o38->atlas2-MDT0000-lwp-OST038e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3f7.ccs.ornl.gov kernel: [801863.509426] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4c1.ccs.ornl.gov kernel: [1915537.860013] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8802d22e3400 x1471904516097584/t0(0) o38->atlas2-MDT0000-lwp-OST0328@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4c1.ccs.ornl.gov kernel: [1915537.944572] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4b6.ccs.ornl.gov kernel: [1915529.982591] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880563ee2c00 x1471904511213644/t0(0) o38->atlas2-MDT0000-lwp-OST03b5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4b6.ccs.ornl.gov kernel: [1915530.067241] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3g7.ccs.ornl.gov kernel: [1914129.928514] Lustre: 13449:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8804764e9c00 x1471904520422328/t0(0) o38->atlas2-MDT0000-lwp-OST0396@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3g7.ccs.ornl.gov kernel: [1914130.015162] Lustre: 13449:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 16:59:37 atlas-oss4b1.ccs.ornl.gov kernel: [1915536.866259] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8804a9dc2400 x1471904507949156/t0(0) o38->atlas2-MDT0000-lwp-OST0290@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4b1.ccs.ornl.gov kernel: [1915536.950910] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3b3.ccs.ornl.gov kernel: [1915525.344098] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8806ee3f7c00 x1471904513820504/t0(0) o38->atlas2-MDT0000-lwp-OST036a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3b3.ccs.ornl.gov kernel: [1915525.436729] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4g8.ccs.ornl.gov kernel: [1915531.531451] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3a5.ccs.ornl.gov kernel: [1915525.937222] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88074201e400 x1471904511738544/t0(0) o38->atlas2-MDT0000-lwp-OST0364@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3a5.ccs.ornl.gov kernel: [1915526.023083] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4f5.ccs.ornl.gov kernel: [1915530.333120] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880cfe509800 x1471904509390996/t0(0) o38->atlas2-MDT0000-lwp-OST0194@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4f5.ccs.ornl.gov kernel: [1915530.419646] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4g3.ccs.ornl.gov kernel: [1915543.546682] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880b782d7c00 x1471904517610296/t0(0) o38->atlas2-MDT0000-lwp-OST022a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4g3.ccs.ornl.gov kernel: [1915543.639756] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3a6.ccs.ornl.gov kernel: [1915535.039818] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8807830a8000 x1471904512394720/t0(0) o38->atlas2-MDT0000-lwp-OST0365@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3a6.ccs.ornl.gov kernel: [1915535.132691] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3i4.ccs.ornl.gov kernel: [1915538.312192] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8806f7306800 x1471904510470456/t0(0) o38->atlas2-MDT0000-lwp-OST03a3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3b5.ccs.ornl.gov kernel: [1915545.452098] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8803142adc00 x1471904514717044/t0(0) o38->atlas2-MDT0000-lwp-OST036c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3b5.ccs.ornl.gov kernel: [1915545.544406] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4g4.ccs.ornl.gov kernel: [1915546.899558] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8806c2bca400 x1471904515530736/t0(0) o38->atlas2-MDT0000-lwp-OST022b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4g7.ccs.ornl.gov kernel: [1915554.754504] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880de5170c00 x1471904508873656/t0(0) o38->atlas2-MDT0000-lwp-OST03de@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4g7.ccs.ornl.gov kernel: [1915554.839322] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3h2.ccs.ornl.gov kernel: [1915535.501033] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88100793cc00 x1471904518760584/t0(0) o38->atlas2-MDT0000-lwp-OST0399@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4g6.ccs.ornl.gov kernel: [1915542.132875] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88029759a800 x1471904507898768/t0(0) o38->atlas2-MDT0000-lwp-OST03dd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4b7.ccs.ornl.gov kernel: [1915526.521782] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88095387e400 x1471904509377368/t0(0) o38->atlas2-MDT0000-lwp-OST03b6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4b7.ccs.ornl.gov kernel: [1915526.606389] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3h4.ccs.ornl.gov kernel: [1915536.240247] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8810643f6800 x1471904509245340/t0(0) o38->atlas2-MDT0000-lwp-OST039b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4f6.ccs.ornl.gov kernel: [1915527.218742] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880935ae8000 x1471904511708636/t0(0) o38->atlas2-MDT0000-lwp-OST0345@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4f6.ccs.ornl.gov kernel: [1915527.321800] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4b8.ccs.ornl.gov kernel: [1915534.324103] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880f1410c400 x1471904512563988/t0(0) o38->atlas2-MDT0000-lwp-OST03b7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4b8.ccs.ornl.gov kernel: [1915534.408951] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4f3.ccs.ornl.gov kernel: [1915527.187933] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8806da73b000 x1471904513977224/t0(0) o38->atlas2-MDT0000-lwp-OST03d2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4f3.ccs.ornl.gov kernel: [1915527.280721] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3i4.ccs.ornl.gov kernel: [1915538.398198] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3e7.ccs.ornl.gov kernel: [1915534.869928] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880a46565000 x1471904508195288/t0(0) o38->atlas2-MDT0000-lwp-OST0386@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3e7.ccs.ornl.gov kernel: [1915534.956143] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3g1.ccs.ornl.gov kernel: [1914122.439999] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880515fb4800 x1471904510299700/t0(0) o38->atlas2-MDT0000-lwp-OST01e0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3g1.ccs.ornl.gov kernel: [1914122.526562] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3g2.ccs.ornl.gov kernel: [1914129.116759] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880202bd1400 x1471904509036200/t0(0) o38->atlas2-MDT0000-lwp-OST0391@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4g4.ccs.ornl.gov kernel: [1915546.984982] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3a2.ccs.ornl.gov kernel: [1915532.016500] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880f72e67800 x1471904510524352/t0(0) o38->atlas2-MDT0000-lwp-OST02d1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3a2.ccs.ornl.gov kernel: [1915532.102363] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4g6.ccs.ornl.gov kernel: [1915542.219826] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3h1.ccs.ornl.gov kernel: [1915521.204886] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88027e780400 x1471904517765976/t0(0) o38->atlas2-MDT0000-lwp-OST0398@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3h2.ccs.ornl.gov kernel: [1915535.586240] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3h4.ccs.ornl.gov kernel: [1915536.327048] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4e8.ccs.ornl.gov kernel: [1915545.455284] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880457992000 x1471904518472872/t0(0) o38->atlas2-MDT0000-lwp-OST03cf@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4d5.ccs.ornl.gov kernel: [1915545.012983] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880500fa4000 x1471904519136900/t0(0) o38->atlas2-MDT0000-lwp-OST03c4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4d5.ccs.ornl.gov kernel: [1915545.101431] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4i8.ccs.ornl.gov kernel: [1915546.201798] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8806b1399800 x1471904520247204/t0(0) o38->atlas2-MDT0000-lwp-OST03ef@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4i8.ccs.ornl.gov kernel: [1915546.288961] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4i5.ccs.ornl.gov kernel: [1915540.466677] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8805c65a1400 x1471904512720472/t0(0) o38->atlas2-MDT0000-lwp-OST023c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3i8.ccs.ornl.gov kernel: [1915546.409502] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8807769de000 x1471904515355548/t0(0) o38->atlas2-MDT0000-lwp-OST03a7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4e3.ccs.ornl.gov kernel: [1915538.320767] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88041c355000 x1471904515523796/t0(0) o38->atlas2-MDT0000-lwp-OST03ca@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4i5.ccs.ornl.gov kernel: [1915540.551511] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4f1.ccs.ornl.gov kernel: [1915536.681998] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8806e261f800 x1471904513371128/t0(0) o38->atlas2-MDT0000-lwp-OST0220@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4e3.ccs.ornl.gov kernel: [1915538.412867] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3i8.ccs.ornl.gov kernel: [1915546.494085] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4f1.ccs.ornl.gov kernel: [1915536.773825] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3g2.ccs.ornl.gov kernel: [1914129.201729] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3d8.ccs.ornl.gov kernel: [1915539.079078] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880f0287e000 x1471904508710740/t0(0) o38->atlas2-MDT0000-lwp-OST025f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3d8.ccs.ornl.gov kernel: [1915539.164358] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4h5.ccs.ornl.gov kernel: [1915535.028588] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8804f6725800 x1471904509313540/t0(0) o38->atlas2-MDT0000-lwp-OST03e4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4h5.ccs.ornl.gov kernel: [1915535.113236] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3h1.ccs.ornl.gov kernel: [1915521.289929] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3f2.ccs.ornl.gov kernel: [1915527.664478] Lustre: 25710:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880fd8626c00 x1471904513672588/t0(0) o38->atlas2-MDT0000-lwp-OST01d9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3a4.ccs.ornl.gov kernel: [1915522.518485] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8804a9cdc000 x1471904512538860/t0(0) o38->atlas2-MDT0000-lwp-OST0363@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3f2.ccs.ornl.gov kernel: [1915527.757323] Lustre: 25710:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3a4.ccs.ornl.gov kernel: [1915522.607067] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4e8.ccs.ornl.gov kernel: [1915545.547827] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4b5.ccs.ornl.gov kernel: [1915550.098210] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8804a127d800 x1471904518365736/t0(0) o38->atlas2-MDT0000-lwp-OST03b4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4b5.ccs.ornl.gov kernel: [1915550.190073] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4i6.ccs.ornl.gov kernel: [1915545.923912] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880712cd7800 x1471904510597656/t0(0) o38->atlas2-MDT0000-lwp-OST035d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3d1.ccs.ornl.gov kernel: [1915525.215235] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8803b52ee800 x1471904515778376/t0(0) o38->atlas2-MDT0000-lwp-OST01c8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4h1.ccs.ornl.gov kernel: [1915526.429721] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8808de56d800 x1471904511603200/t0(0) o38->atlas2-MDT0000-lwp-OST0350@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4h1.ccs.ornl.gov kernel: [1915526.514326] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3e8.ccs.ornl.gov kernel: [1915532.672193] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880437c8ac00 x1471904515139660/t0(0) o38->atlas2-MDT0000-lwp-OST0387@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3e8.ccs.ornl.gov kernel: [1915532.758161] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3a3.ccs.ornl.gov kernel: [1915529.795376] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880da85a6800 x1471904526379988/t0(0) o38->atlas2-MDT0000-lwp-OST02d2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4f4.ccs.ornl.gov kernel: [1915533.010326] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88034d35b400 x1471904516980336/t0(0) o38->atlas2-MDT0000-lwp-OST0343@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4f4.ccs.ornl.gov kernel: [1915533.102384] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3b2.ccs.ornl.gov kernel: [1915532.555951] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8809562cb800 x1471904511917072/t0(0) o38->atlas2-MDT0000-lwp-OST0249@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3b2.ccs.ornl.gov kernel: [1915532.643398] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3e5.ccs.ornl.gov kernel: [1915533.812825] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8809ed2e3000 x1471904510654880/t0(0) o38->atlas2-MDT0000-lwp-OST0384@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3e5.ccs.ornl.gov kernel: [1915533.897398] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3b6.ccs.ornl.gov kernel: [1915530.781967] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88043d354400 x1471904512275452/t0(0) o38->atlas2-MDT0000-lwp-OST02dd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3b6.ccs.ornl.gov kernel: [1915530.885185] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4a6.ccs.ornl.gov kernel: [1915534.630806] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880536474400 x1471904512502848/t0(0) o38->atlas2-MDT0000-lwp-OST03ad@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4a6.ccs.ornl.gov kernel: [1915534.723758] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4e5.ccs.ornl.gov kernel: [1915533.640260] Lustre: 24644:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88086c619c00 x1471904515279276/t0(0) o38->atlas2-MDT0000-lwp-OST02ac@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4e5.ccs.ornl.gov kernel: [1915533.743706] Lustre: 24644:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3g4.ccs.ornl.gov kernel: [1914143.071331] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880d35f63c00 x1471904510791424/t0(0) o38->atlas2-MDT0000-lwp-OST0273@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3g4.ccs.ornl.gov kernel: [1914143.156338] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4d6.ccs.ornl.gov kernel: [1915532.661371] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880238071000 x1471904519610492/t0(0) o38->atlas2-MDT0000-lwp-OST0335@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3c5.ccs.ornl.gov kernel: [1915539.187051] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880cbb4c5c00 x1471904512466200/t0(0) o38->atlas2-MDT0000-lwp-OST0374@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4i6.ccs.ornl.gov kernel: [1915546.011225] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3d1.ccs.ornl.gov kernel: [1915525.303188] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4g1.ccs.ornl.gov kernel: [1915529.657707] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8807a09da400 x1471904508970472/t0(0) o38->atlas2-MDT0000-lwp-OST03d8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4g1.ccs.ornl.gov kernel: [1915529.748664] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3c2.ccs.ornl.gov kernel: [1915534.628085] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88076bb93c00 x1471904511648012/t0(0) o38->atlas2-MDT0000-lwp-OST0371@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3c2.ccs.ornl.gov kernel: [1915534.720490] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3a3.ccs.ornl.gov kernel: [1915529.882045] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3f3.ccs.ornl.gov kernel: [1915546.111332] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88006f3fbc00 x1471904510688164/t0(0) o38->atlas2-MDT0000-lwp-OST038a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3c7.ccs.ornl.gov kernel: [1915533.465796] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88034cd6a800 x1471904510248752/t0(0) o38->atlas2-MDT0000-lwp-OST0376@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3c7.ccs.ornl.gov kernel: [1915533.550932] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3c6.ccs.ornl.gov kernel: [1915524.847916] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880bcafae400 x1471904508186952/t0(0) o38->atlas2-MDT0000-lwp-OST0375@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3c6.ccs.ornl.gov kernel: [1915524.940592] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3e4.ccs.ornl.gov kernel: [1915546.845907] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8806f8066400 x1471904508538088/t0(0) o38->atlas2-MDT0000-lwp-OST01d3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3e4.ccs.ornl.gov kernel: [1915546.930584] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3d3.ccs.ornl.gov kernel: [1915539.430190] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8806e4079000 x1471904510125796/t0(0) o38->atlas2-MDT0000-lwp-OST037a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4d6.ccs.ornl.gov kernel: [1915532.747026] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4a3.ccs.ornl.gov kernel: [1915548.567876] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880721a6f400 x1471904514688968/t0(0) o38->atlas2-MDT0000-lwp-OST03aa@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4a3.ccs.ornl.gov kernel: [1915548.656104] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4c6.ccs.ornl.gov kernel: [1915533.520982] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880a7a5c9400 x1471904517937296/t0(0) o38->atlas2-MDT0000-lwp-OST032d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4c6.ccs.ornl.gov kernel: [1915533.609224] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3c8.ccs.ornl.gov kernel: [1915537.659525] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880e9bd72800 x1471904507928116/t0(0) o38->atlas2-MDT0000-lwp-OST02e7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3c8.ccs.ornl.gov kernel: [1915537.745117] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3c5.ccs.ornl.gov kernel: [1915539.271746] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4a7.ccs.ornl.gov kernel: [1915544.272475] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8806b08ef800 x1471904510936096/t0(0) o38->atlas2-MDT0000-lwp-OST03ae@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4a1.ccs.ornl.gov kernel: [1915530.565147] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880dedc9e400 x1471904510345880/t0(0) o38->atlas2-MDT0000-lwp-OST03a8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4a1.ccs.ornl.gov kernel: [1915530.654067] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3f3.ccs.ornl.gov kernel: [1915546.204740] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4e6.ccs.ornl.gov kernel: [1915548.373426] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8805aaf15c00 x1471904508408188/t0(0) o38->atlas2-MDT0000-lwp-OST033d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4e6.ccs.ornl.gov kernel: [1915548.458891] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3e2.ccs.ornl.gov kernel: [1915532.007073] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8803c72b9c00 x1471904513179268/t0(0) o38->atlas2-MDT0000-lwp-OST0381@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3e2.ccs.ornl.gov kernel: [1915532.099378] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3h6.ccs.ornl.gov kernel: [1915533.003108] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880cc0c40000 x1471904512216600/t0(0) o38->atlas2-MDT0000-lwp-OST039d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3h6.ccs.ornl.gov kernel: [1915533.089373] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3d2.ccs.ornl.gov kernel: [1915534.348049] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8809fe4c6400 x1471904511214712/t0(0) o38->atlas2-MDT0000-lwp-OST01c9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3d2.ccs.ornl.gov kernel: [1915534.433053] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3d3.ccs.ornl.gov kernel: [1915539.515011] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3i6.ccs.ornl.gov kernel: [1915531.496807] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880870ba8800 x1471904509634080/t0(0) o38->atlas2-MDT0000-lwp-OST03a5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3i6.ccs.ornl.gov kernel: [1915531.583009] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3c4.ccs.ornl.gov kernel: [1915527.930327] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8803baf91000 x1471904511832984/t0(0) o38->atlas2-MDT0000-lwp-OST0253@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3c4.ccs.ornl.gov kernel: [1915528.015308] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4a7.ccs.ornl.gov kernel: [1915544.359318] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4d1.ccs.ornl.gov kernel: [1915528.678206] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880b62109000 x1471904516811784/t0(0) o38->atlas2-MDT0000-lwp-OST03c0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4d1.ccs.ornl.gov kernel: [1915528.762808] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3g3.ccs.ornl.gov kernel: [1914130.032751] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88054cc68800 x1471904516976836/t0(0) o38->atlas2-MDT0000-lwp-OST0392@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3g3.ccs.ornl.gov kernel: [1914130.125877] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3a7.ccs.ornl.gov kernel: [1915530.782620] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880474fb8800 x1471904518655704/t0(0) o38->atlas2-MDT0000-lwp-OST0366@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3a7.ccs.ornl.gov kernel: [1915530.869352] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3i3.ccs.ornl.gov kernel: [1915528.127837] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880b60235800 x1471904518169348/t0(0) o38->atlas2-MDT0000-lwp-OST03a2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3i3.ccs.ornl.gov kernel: [1915528.221150] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3c3.ccs.ornl.gov kernel: [1915525.358163] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8801e137ac00 x1471904511051520/t0(0) o38->atlas2-MDT0000-lwp-OST0372@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3c3.ccs.ornl.gov kernel: [1915525.443058] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3d7.ccs.ornl.gov kernel: [1915533.538867] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880cd0ad4400 x1471904512149524/t0(0) o38->atlas2-MDT0000-lwp-OST037e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3d7.ccs.ornl.gov kernel: [1915533.623530] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4i1.ccs.ornl.gov kernel: [1915532.788464] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8807060f4400 x1471904513319796/t0(0) o38->atlas2-MDT0000-lwp-OST0238@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4i1.ccs.ornl.gov kernel: [1915532.873493] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3g6.ccs.ornl.gov kernel: [1914141.034286] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880c8ad6e400 x1471904523473904/t0(0) o38->atlas2-MDT0000-lwp-OST0395@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3g6.ccs.ornl.gov kernel: [1914141.127213] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3e6.ccs.ornl.gov kernel: [1915530.241041] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880e01e36000 x1471904520534712/t0(0) o38->atlas2-MDT0000-lwp-OST0385@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3e6.ccs.ornl.gov kernel: [1915530.326990] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4a8.ccs.ornl.gov kernel: [1915538.718127] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88100b1ca400 x1471904507750840/t0(0) o38->atlas2-MDT0000-lwp-OST03af@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4a8.ccs.ornl.gov kernel: [1915538.803201] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4b4.ccs.ornl.gov kernel: [1915530.782994] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88047c434800 x1471904528872084/t0(0) o38->atlas2-MDT0000-lwp-OST0293@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4b4.ccs.ornl.gov kernel: [1915530.870224] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4h3.ccs.ornl.gov kernel: [1915545.354942] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880f07a2e800 x1471904509650920/t0(0) o38->atlas2-MDT0000-lwp-OST03e2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3f1.ccs.ornl.gov kernel: [1915536.511108] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88018e510800 x1471904508988504/t0(0) o38->atlas2-MDT0000-lwp-OST0388@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3f1.ccs.ornl.gov kernel: [1915536.602403] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3b7.ccs.ornl.gov kernel: [1915537.569773] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8805f387bc00 x1471904519392716/t0(0) o38->atlas2-MDT0000-lwp-OST036e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3b7.ccs.ornl.gov kernel: [1915537.656070] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4f8.ccs.ornl.gov kernel: [1915531.928977] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8802b0131000 x1471904511806092/t0(0) o38->atlas2-MDT0000-lwp-OST0227@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4f8.ccs.ornl.gov kernel: [1915532.021749] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3b1.ccs.ornl.gov kernel: [1915539.342303] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8804a922c400 x1471904516110824/t0(0) o38->atlas2-MDT0000-lwp-OST02d8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3b1.ccs.ornl.gov kernel: [1915539.435402] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4h3.ccs.ornl.gov kernel: [1915545.447831] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3i1.ccs.ornl.gov kernel: [1915532.845953] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880785c9f400 x1471904519755480/t0(0) o38->atlas2-MDT0000-lwp-OST03a0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3i1.ccs.ornl.gov kernel: [1915532.939147] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss4d8.ccs.ornl.gov kernel: [1915545.759469] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8806b9f28000 x1471904516223068/t0(0) o38->atlas2-MDT0000-lwp-OST03c7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss4d8.ccs.ornl.gov kernel: [1915545.852291] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3b8.ccs.ornl.gov kernel: [1915546.885950] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff88100677bc00 x1471904516406852/t0(0) o38->atlas2-MDT0000-lwp-OST036f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3d5.ccs.ornl.gov kernel: [1915536.242922] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8802495f7c00 x1471904509642104/t0(0) o38->atlas2-MDT0000-lwp-OST037c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3d5.ccs.ornl.gov kernel: [1915536.328227] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3b8.ccs.ornl.gov kernel: [1915546.971413] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3b4.ccs.ornl.gov kernel: [1915544.410671] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff880e0f6ccc00 x1471904510548628/t0(0) o38->atlas2-MDT0000-lwp-OST02db@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3b4.ccs.ornl.gov kernel: [1915544.495487] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 16:59:37 atlas-oss3i2.ccs.ornl.gov kernel: [1915538.954763] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630497/real 1405630497] req@ffff8803f25d8c00 x1471904514016336/t0(0) o38->atlas2-MDT0000-lwp-OST03a1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 16:59:37 atlas-oss3i2.ccs.ornl.gov kernel: [1915539.040010] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:00:23 atlas-oss3i5.ccs.ornl.gov kernel: [701886.786203] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630543/real 1405630543] req@ffff8807926b7400 x1473175721933408/t0(0) o38->atlas2-MDT0000-lwp-OST0314@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630823 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:00:23 atlas-oss3i5.ccs.ornl.gov kernel: [701886.872540] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:00:23 atlas-oss3a8.ccs.ornl.gov kernel: [701887.753775] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630543/real 1405630543] req@ffff88022a0f9400 x1473175720194988/t0(0) o38->atlas2-MDT0000-lwp-OST02d7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630823 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:00:23 atlas-oss3a8.ccs.ornl.gov kernel: [701887.846641] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:00:24 atlas-oss3d4.ccs.ornl.gov kernel: [701887.269885] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630543/real 1405630543] req@ffff880870a6a400 x1473175723812788/t0(0) o38->atlas2-MDT0000-lwp-OST037b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405630823 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:00:24 atlas-oss3d4.ccs.ornl.gov kernel: [701887.354642] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4g5.ccs.ornl.gov kernel: [1915832.339528] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630796/real 1405630796] req@ffff8802d9884000 x1471904510456848/t0(0) o38->atlas2-MDT0000-lwp-OST022c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4e4.ccs.ornl.gov kernel: [1915840.931121] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630796/real 1405630796] req@ffff8806f7414000 x1471904515374444/t0(0) o38->atlas2-MDT0000-lwp-OST03cb@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4e4.ccs.ornl.gov kernel: [1915841.016995] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4g2.ccs.ornl.gov kernel: [1915843.012848] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630796/real 1405630796] req@ffff880f93829800 x1471904509229092/t0(0) o38->atlas2-MDT0000-lwp-OST03d9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4g2.ccs.ornl.gov kernel: [1915843.099649] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4h7.ccs.ornl.gov kernel: [1915840.534383] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630796/real 1405630796] req@ffff8807168a5c00 x1471904512615260/t0(0) o38->atlas2-MDT0000-lwp-OST02c6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4h7.ccs.ornl.gov kernel: [1915840.627084] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4g5.ccs.ornl.gov kernel: [1915832.423941] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4d2.ccs.ornl.gov kernel: [1915837.351829] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630796/real 1405630796] req@ffff88018f9bf400 x1471904512221076/t0(0) o38->atlas2-MDT0000-lwp-OST0331@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4i7.ccs.ornl.gov kernel: [1915845.543281] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630796/real 1405630796] req@ffff880c1bb4b800 x1471904511375228/t0(0) o38->atlas2-MDT0000-lwp-OST035e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4d2.ccs.ornl.gov kernel: [1915837.444935] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4i7.ccs.ornl.gov kernel: [1915845.628655] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4h6.ccs.ornl.gov kernel: [1915831.057857] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630796/real 1405630796] req@ffff880e46701400 x1471904515532220/t0(0) o38->atlas2-MDT0000-lwp-OST03e5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4h6.ccs.ornl.gov kernel: [1915831.147646] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4i2.ccs.ornl.gov kernel: [1915855.131309] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630796/real 1405630796] req@ffff88079ef9c400 x1471904514634644/t0(0) o38->atlas2-MDT0000-lwp-OST0359@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4i2.ccs.ornl.gov kernel: [1915855.217721] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4b3.ccs.ornl.gov kernel: [1915826.570530] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630796/real 1405630796] req@ffff88030fae5000 x1471904509180640/t0(0) o38->atlas2-MDT0000-lwp-OST03b2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4f7.ccs.ornl.gov kernel: [1915831.341558] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630796/real 1405630796] req@ffff8801931eb800 x1471904510455156/t0(0) o38->atlas2-MDT0000-lwp-OST02b6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4f7.ccs.ornl.gov kernel: [1915831.430089] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4h2.ccs.ornl.gov kernel: [1915853.505913] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630796/real 1405630796] req@ffff880816af8c00 x1471904508411256/t0(0) o38->atlas2-MDT0000-lwp-OST03e1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4h2.ccs.ornl.gov kernel: [1915853.598842] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4b3.ccs.ornl.gov kernel: [1915826.660468] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3e3.ccs.ornl.gov kernel: [1915828.995587] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8801e607cc00 x1471904510468564/t0(0) o38->atlas2-MDT0000-lwp-OST02f2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3e3.ccs.ornl.gov kernel: [1915829.080057] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4i3.ccs.ornl.gov kernel: [1915824.558211] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8808eef3f400 x1471904523159700/t0(0) o38->atlas2-MDT0000-lwp-OST03ea@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4d7.ccs.ornl.gov kernel: [1915833.031140] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880d04e16400 x1471904514720096/t0(0) o38->atlas2-MDT0000-lwp-OST0336@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4d7.ccs.ornl.gov kernel: [1915833.119705] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3h5.ccs.ornl.gov kernel: [1915852.044898] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880317f29800 x1471904507806704/t0(0) o38->atlas2-MDT0000-lwp-OST039c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3h5.ccs.ornl.gov kernel: [1915852.133420] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3f6.ccs.ornl.gov kernel: [802536.925776] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880af7b64000 x1473076856558696/t0(0) o38->atlas2-MDT0000-lwp-OST02fd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3f6.ccs.ornl.gov kernel: [802537.012617] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4i4.ccs.ornl.gov kernel: [1915837.245854] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880a8375fc00 x1471904516513548/t0(0) o38->atlas2-MDT0000-lwp-OST03eb@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4i4.ccs.ornl.gov kernel: [1915837.335523] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3c1.ccs.ornl.gov kernel: [1915834.478859] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88077dd10c00 x1471904511393084/t0(0) o38->atlas2-MDT0000-lwp-OST0370@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3c1.ccs.ornl.gov kernel: [1915834.566145] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3i7.ccs.ornl.gov kernel: [1915829.498847] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880a63f2f000 x1471904508389412/t0(0) o38->atlas2-MDT0000-lwp-OST03a6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3i7.ccs.ornl.gov kernel: [1915829.586012] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4f2.ccs.ornl.gov kernel: [1915835.921294] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880f59b6c800 x1471904508910792/t0(0) o38->atlas2-MDT0000-lwp-OST02b1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4f2.ccs.ornl.gov kernel: [1915836.008915] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4d3.ccs.ornl.gov kernel: [1915828.829329] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88097dd57400 x1471904516978368/t0(0) o38->atlas2-MDT0000-lwp-OST0212@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4d3.ccs.ornl.gov kernel: [1915828.917149] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4c4.ccs.ornl.gov kernel: [1915844.227244] Lustre: 25786:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880ef5ab1800 x1471904510981176/t0(0) o38->atlas2-MDT0000-lwp-OST03bb@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4c4.ccs.ornl.gov kernel: [1915844.313171] Lustre: 25786:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4i3.ccs.ornl.gov kernel: [1915824.645165] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4c5.ccs.ornl.gov kernel: [1915828.114016] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8805b2815c00 x1471904510171340/t0(0) o38->atlas2-MDT0000-lwp-OST032c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4c5.ccs.ornl.gov kernel: [1915828.202364] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4h4.ccs.ornl.gov kernel: [1915851.776461] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8804ce25c800 x1471904512334464/t0(0) o38->atlas2-MDT0000-lwp-OST0233@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4c3.ccs.ornl.gov kernel: [1915829.155509] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880f22a75000 x1471904512246192/t0(0) o38->atlas2-MDT0000-lwp-OST032a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4c3.ccs.ornl.gov kernel: [1915829.242869] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3h8.ccs.ornl.gov kernel: [1915829.248140] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8803e2a9dc00 x1471904514772936/t0(0) o38->atlas2-MDT0000-lwp-OST039f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3h8.ccs.ornl.gov kernel: [1915829.332985] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3f4.ccs.ornl.gov kernel: [1915834.554349] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880d53410400 x1471904507778584/t0(0) o38->atlas2-MDT0000-lwp-OST026b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3f4.ccs.ornl.gov kernel: [1915834.639957] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4a4.ccs.ornl.gov kernel: [1915833.936689] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880435951c00 x1471904509591200/t0(0) o38->atlas2-MDT0000-lwp-OST01fb@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4a4.ccs.ornl.gov kernel: [1915834.021799] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3d6.ccs.ornl.gov kernel: [1915846.771467] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880eaf3b7c00 x1471904511183496/t0(0) o38->atlas2-MDT0000-lwp-OST037d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3d6.ccs.ornl.gov kernel: [1915846.861298] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3g8.ccs.ornl.gov kernel: [1914430.428746] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88031f9abc00 x1471904513602532/t0(0) o38->atlas2-MDT0000-lwp-OST0397@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3g8.ccs.ornl.gov kernel: [1914430.521615] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4a5.ccs.ornl.gov kernel: [1915851.357056] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880a22ade800 x1471904519454260/t0(0) o38->atlas2-MDT0000-lwp-OST03ac@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4a5.ccs.ornl.gov kernel: [1915851.446770] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4h4.ccs.ornl.gov kernel: [1915851.866250] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3e1.ccs.ornl.gov kernel: [1915835.151185] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88049af9b000 x1471904509694460/t0(0) o38->atlas2-MDT0000-lwp-OST02f0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3e1.ccs.ornl.gov kernel: [1915835.236501] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4e7.ccs.ornl.gov kernel: [1915829.815909] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880871abc000 x1471904522621624/t0(0) o38->atlas2-MDT0000-lwp-OST033e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4e7.ccs.ornl.gov kernel: [1915829.900897] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3f5.ccs.ornl.gov kernel: [802524.306445] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880972235400 x1473076856406408/t0(0) o38->atlas2-MDT0000-lwp-OST038c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4a2.ccs.ornl.gov kernel: [1915847.037783] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8807500f4800 x1471904510895816/t0(0) o38->atlas2-MDT0000-lwp-OST0289@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4a2.ccs.ornl.gov kernel: [1915847.126566] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4c8.ccs.ornl.gov kernel: [1915824.266820] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880fd76f8400 x1471904512197032/t0(0) o38->atlas2-MDT0000-lwp-OST03bf@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4c8.ccs.ornl.gov kernel: [1915824.358918] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3f8.ccs.ornl.gov kernel: [802534.810828] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880b3f288400 x1473076861321056/t0(0) o38->atlas2-MDT0000-lwp-OST038f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3g5.ccs.ornl.gov kernel: [1914434.009951] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88044dcab800 x1471904518249188/t0(0) o38->atlas2-MDT0000-lwp-OST01e4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4c2.ccs.ornl.gov kernel: [1915836.357752] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880ef2ecf400 x1471904508570780/t0(0) o38->atlas2-MDT0000-lwp-OST03b9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4c2.ccs.ornl.gov kernel: [1915836.450941] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3f5.ccs.ornl.gov kernel: [802524.392433] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4e1.ccs.ornl.gov kernel: [1915830.308809] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff881043656c00 x1471904515144868/t0(0) o38->atlas2-MDT0000-lwp-OST03c8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3f8.ccs.ornl.gov kernel: [802534.899516] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4e1.ccs.ornl.gov kernel: [1915830.394687] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3g5.ccs.ornl.gov kernel: [1914434.097281] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4g8.ccs.ornl.gov kernel: [1915831.548599] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88082b1cb800 x1471904521746940/t0(0) o38->atlas2-MDT0000-lwp-OST03df@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4g8.ccs.ornl.gov kernel: [1915831.634009] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4d4.ccs.ornl.gov kernel: [1915852.801940] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880871ccf400 x1471904513679268/t0(0) o38->atlas2-MDT0000-lwp-OST03c3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4d4.ccs.ornl.gov kernel: [1915852.889114] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3f7.ccs.ornl.gov kernel: [802163.528397] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8803c1289400 x1473076863306304/t0(0) o38->atlas2-MDT0000-lwp-OST038e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3f7.ccs.ornl.gov kernel: [802163.613275] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3h3.ccs.ornl.gov kernel: [1915828.053708] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88065b2ce000 x1471904512413708/t0(0) o38->atlas2-MDT0000-lwp-OST01ea@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3h7.ccs.ornl.gov kernel: [1915839.179206] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88014cfa4c00 x1471904510333048/t0(0) o38->atlas2-MDT0000-lwp-OST027e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3h7.ccs.ornl.gov kernel: [1915839.266645] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4e2.ccs.ornl.gov kernel: [1915846.272620] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8809f16b6c00 x1471904514146444/t0(0) o38->atlas2-MDT0000-lwp-OST0219@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3a1.ccs.ornl.gov kernel: [1915827.284652] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880d35333400 x1471904525931876/t0(0) o38->atlas2-MDT0000-lwp-OST0360@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4e2.ccs.ornl.gov kernel: [1915846.358584] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3a1.ccs.ornl.gov kernel: [1915827.375612] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4h8.ccs.ornl.gov kernel: [1915823.408451] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8803a98e0400 x1471904514267304/t0(0) o38->atlas2-MDT0000-lwp-OST02c7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4c1.ccs.ornl.gov kernel: [1915837.971880] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880251d4c400 x1471904516097620/t0(0) o38->atlas2-MDT0000-lwp-OST0328@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4c1.ccs.ornl.gov kernel: [1915838.058292] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4b6.ccs.ornl.gov kernel: [1915830.091543] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88081dac7800 x1471904511213680/t0(0) o38->atlas2-MDT0000-lwp-OST03b5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4b6.ccs.ornl.gov kernel: [1915830.178802] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4b1.ccs.ornl.gov kernel: [1915836.978010] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880180010800 x1471904507949204/t0(0) o38->atlas2-MDT0000-lwp-OST03b0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4b1.ccs.ornl.gov kernel: [1915837.063157] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3b3.ccs.ornl.gov kernel: [1915825.453941] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88078124a000 x1471904513820536/t0(0) o38->atlas2-MDT0000-lwp-OST02da@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3b3.ccs.ornl.gov kernel: [1915825.546906] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3h3.ccs.ornl.gov kernel: [1915828.143444] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4h8.ccs.ornl.gov kernel: [1915823.494940] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4g3.ccs.ornl.gov kernel: [1915843.658965] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880e6d4d6400 x1471904517610344/t0(0) o38->atlas2-MDT0000-lwp-OST03da@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4g3.ccs.ornl.gov kernel: [1915843.744574] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3a5.ccs.ornl.gov kernel: [1915826.047671] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880770557c00 x1471904511738576/t0(0) o38->atlas2-MDT0000-lwp-OST02d4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3a5.ccs.ornl.gov kernel: [1915826.135823] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3a6.ccs.ornl.gov kernel: [1915835.151097] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880772d32400 x1471904512394756/t0(0) o38->atlas2-MDT0000-lwp-OST0365@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3a6.ccs.ornl.gov kernel: [1915835.236003] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4f5.ccs.ornl.gov kernel: [1915830.444407] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880d7939c000 x1471904509391032/t0(0) o38->atlas2-MDT0000-lwp-OST0194@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4f5.ccs.ornl.gov kernel: [1915830.542134] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3i4.ccs.ornl.gov kernel: [1915838.423830] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88014f252800 x1471904510470492/t0(0) o38->atlas2-MDT0000-lwp-OST03a3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3b5.ccs.ornl.gov kernel: [1915845.564889] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8808552e2400 x1471904514717080/t0(0) o38->atlas2-MDT0000-lwp-OST036c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3b5.ccs.ornl.gov kernel: [1915845.657745] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4g4.ccs.ornl.gov kernel: [1915847.012775] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88056895fc00 x1471904515530776/t0(0) o38->atlas2-MDT0000-lwp-OST02bb@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4g4.ccs.ornl.gov kernel: [1915847.099510] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4f6.ccs.ornl.gov kernel: [1915827.334410] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880f857b8c00 x1471904511708672/t0(0) o38->atlas2-MDT0000-lwp-OST0345@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4f6.ccs.ornl.gov kernel: [1915827.424006] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4g7.ccs.ornl.gov kernel: [1915854.868428] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880f535ea800 x1471904508873692/t0(0) o38->atlas2-MDT0000-lwp-OST03de@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4g7.ccs.ornl.gov kernel: [1915854.953993] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4b7.ccs.ornl.gov kernel: [1915826.632402] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880b5fc46400 x1471904509377404/t0(0) o38->atlas2-MDT0000-lwp-OST03b6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3h2.ccs.ornl.gov kernel: [1915835.612388] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880b95768400 x1471904518760620/t0(0) o38->atlas2-MDT0000-lwp-OST0399@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3h2.ccs.ornl.gov kernel: [1915835.698918] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4f3.ccs.ornl.gov kernel: [1915827.297603] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88086fb65c00 x1471904513977248/t0(0) o38->atlas2-MDT0000-lwp-OST02b2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4f3.ccs.ornl.gov kernel: [1915827.382541] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4g6.ccs.ornl.gov kernel: [1915842.245858] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8805a10d6400 x1471904507898804/t0(0) o38->atlas2-MDT0000-lwp-OST03dd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4g6.ccs.ornl.gov kernel: [1915842.335053] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3h4.ccs.ornl.gov kernel: [1915836.351792] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880decd58000 x1471904509245376/t0(0) o38->atlas2-MDT0000-lwp-OST039b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3h4.ccs.ornl.gov kernel: [1915836.440535] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4b8.ccs.ornl.gov kernel: [1915834.436326] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880bdb838800 x1471904512564024/t0(0) o38->atlas2-MDT0000-lwp-OST03b7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4b8.ccs.ornl.gov kernel: [1915834.521007] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3i4.ccs.ornl.gov kernel: [1915838.511538] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3e7.ccs.ornl.gov kernel: [1915834.982020] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880f33c9d400 x1471904508195320/t0(0) o38->atlas2-MDT0000-lwp-OST02f6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3e7.ccs.ornl.gov kernel: [1915835.069298] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3g1.ccs.ornl.gov kernel: [1914422.549991] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880395ff3400 x1471904510299744/t0(0) o38->atlas2-MDT0000-lwp-OST0390@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3g1.ccs.ornl.gov kernel: [1914422.638725] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3g2.ccs.ornl.gov kernel: [1914429.228850] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8807e6eca800 x1471904509036228/t0(0) o38->atlas2-MDT0000-lwp-OST0271@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3g2.ccs.ornl.gov kernel: [1914429.314328] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3a2.ccs.ornl.gov kernel: [1915832.127310] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff881024e4dc00 x1471904510524392/t0(0) o38->atlas2-MDT0000-lwp-OST0361@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3a2.ccs.ornl.gov kernel: [1915832.215155] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4b7.ccs.ornl.gov kernel: [1915826.718000] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3h1.ccs.ornl.gov kernel: [1915821.313049] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8805b8678400 x1471904517766012/t0(0) o38->atlas2-MDT0000-lwp-OST0398@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3h1.ccs.ornl.gov kernel: [1915821.400429] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4e3.ccs.ornl.gov kernel: [1915838.430206] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88024eac5c00 x1471904515523832/t0(0) o38->atlas2-MDT0000-lwp-OST03ca@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4e3.ccs.ornl.gov kernel: [1915838.515438] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4e8.ccs.ornl.gov kernel: [1915845.567665] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88035818d400 x1471904518472908/t0(0) o38->atlas2-MDT0000-lwp-OST03cf@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4e8.ccs.ornl.gov kernel: [1915845.661098] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4d5.ccs.ornl.gov kernel: [1915845.125227] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8803870efc00 x1471904519136936/t0(0) o38->atlas2-MDT0000-lwp-OST03c4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4d5.ccs.ornl.gov kernel: [1915845.216201] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4i8.ccs.ornl.gov kernel: [1915846.314189] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8807ba043800 x1471904520247240/t0(0) o38->atlas2-MDT0000-lwp-OST03ef@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4i8.ccs.ornl.gov kernel: [1915846.403596] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4f1.ccs.ornl.gov kernel: [1915836.792282] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8805e4b3b800 x1471904513371164/t0(0) o38->atlas2-MDT0000-lwp-OST0220@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4f1.ccs.ornl.gov kernel: [1915836.885445] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3i8.ccs.ornl.gov kernel: [1915846.522631] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8806a3b92400 x1471904515355584/t0(0) o38->atlas2-MDT0000-lwp-OST03a7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3i8.ccs.ornl.gov kernel: [1915846.609016] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3d8.ccs.ornl.gov kernel: [1915839.191388] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880ff43c4c00 x1471904508710784/t0(0) o38->atlas2-MDT0000-lwp-OST037f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3d8.ccs.ornl.gov kernel: [1915839.278283] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4i5.ccs.ornl.gov kernel: [1915840.578591] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8803a8364000 x1471904512720520/t0(0) o38->atlas2-MDT0000-lwp-OST03ec@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4i5.ccs.ornl.gov kernel: [1915840.674654] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4h5.ccs.ornl.gov kernel: [1915835.141458] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8804528a8800 x1471904509313576/t0(0) o38->atlas2-MDT0000-lwp-OST03e4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4h5.ccs.ornl.gov kernel: [1915835.226325] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3f2.ccs.ornl.gov kernel: [1915827.774639] Lustre: 25710:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880dad365c00 x1471904513672636/t0(0) o38->atlas2-MDT0000-lwp-OST0389@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3f2.ccs.ornl.gov kernel: [1915827.859181] Lustre: 25710:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4c7.ccs.ornl.gov kernel: [1915832.520038] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880509782000 x1471904514425716/t0(0) o38->atlas2-MDT0000-lwp-OST020e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4c7.ccs.ornl.gov kernel: [1915832.606793] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:04:37 atlas-oss4b5.ccs.ornl.gov kernel: [1915850.212910] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8803e8b9f000 x1471904518365760/t0(0) o38->atlas2-MDT0000-lwp-OST0204@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4b5.ccs.ornl.gov kernel: [1915850.304066] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4i6.ccs.ornl.gov kernel: [1915846.036852] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880734fc4000 x1471904510597684/t0(0) o38->atlas2-MDT0000-lwp-OST02cd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4i6.ccs.ornl.gov kernel: [1915846.125822] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3a4.ccs.ornl.gov kernel: [1915822.628045] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880809bac800 x1471904512538884/t0(0) o38->atlas2-MDT0000-lwp-OST01b3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3a4.ccs.ornl.gov kernel: [1915822.729408] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3d1.ccs.ornl.gov kernel: [1915825.325113] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8807262d4800 x1471904515778412/t0(0) o38->atlas2-MDT0000-lwp-OST01c8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3d1.ccs.ornl.gov kernel: [1915825.415570] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3b6.ccs.ornl.gov kernel: [1915830.892391] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88039050cc00 x1471904512275488/t0(0) o38->atlas2-MDT0000-lwp-OST02dd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3b6.ccs.ornl.gov kernel: [1915830.977775] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3e8.ccs.ornl.gov kernel: [1915832.782711] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8802d96a8800 x1471904515139696/t0(0) o38->atlas2-MDT0000-lwp-OST0387@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3e8.ccs.ornl.gov kernel: [1915832.870778] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4h1.ccs.ornl.gov kernel: [1915826.541500] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880c84f8ec00 x1471904511603236/t0(0) o38->atlas2-MDT0000-lwp-OST0350@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4f4.ccs.ornl.gov kernel: [1915833.121211] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88055842a800 x1471904516980372/t0(0) o38->atlas2-MDT0000-lwp-OST0343@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4f4.ccs.ornl.gov kernel: [1915833.213772] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3a3.ccs.ornl.gov kernel: [1915829.905401] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880aaadf1400 x1471904526380028/t0(0) o38->atlas2-MDT0000-lwp-OST0362@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3a3.ccs.ornl.gov kernel: [1915829.994906] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3b2.ccs.ornl.gov kernel: [1915832.667917] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880f3ed4b000 x1471904511917116/t0(0) o38->atlas2-MDT0000-lwp-OST0369@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3b2.ccs.ornl.gov kernel: [1915832.757012] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3e5.ccs.ornl.gov kernel: [1915833.922541] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8808ba41dc00 x1471904510654916/t0(0) o38->atlas2-MDT0000-lwp-OST0384@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3e5.ccs.ornl.gov kernel: [1915834.009066] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4a6.ccs.ornl.gov kernel: [1915834.742133] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880292bbdc00 x1471904512502884/t0(0) o38->atlas2-MDT0000-lwp-OST03ad@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4a6.ccs.ornl.gov kernel: [1915834.835503] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3c2.ccs.ornl.gov kernel: [1915834.738245] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8805bf2ed800 x1471904511648048/t0(0) o38->atlas2-MDT0000-lwp-OST0371@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3g4.ccs.ornl.gov kernel: [1914443.184809] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8808d34f0c00 x1471904510791472/t0(0) o38->atlas2-MDT0000-lwp-OST0393@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3c2.ccs.ornl.gov kernel: [1915834.822962] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3g4.ccs.ornl.gov kernel: [1914443.271535] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4d6.ccs.ornl.gov kernel: [1915832.772411] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88059957c800 x1471904519610528/t0(0) o38->atlas2-MDT0000-lwp-OST0335@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4d6.ccs.ornl.gov kernel: [1915832.859618] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3c5.ccs.ornl.gov kernel: [1915839.298107] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880e3d460000 x1471904512466236/t0(0) o38->atlas2-MDT0000-lwp-OST0374@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3c5.ccs.ornl.gov kernel: [1915839.385574] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3f3.ccs.ornl.gov kernel: [1915846.223833] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88070a7e5800 x1471904510688200/t0(0) o38->atlas2-MDT0000-lwp-OST038a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4g1.ccs.ornl.gov kernel: [1915829.767402] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88024ce04800 x1471904508970508/t0(0) o38->atlas2-MDT0000-lwp-OST03d8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4g1.ccs.ornl.gov kernel: [1915829.859937] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4h1.ccs.ornl.gov kernel: [1915826.634329] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3c6.ccs.ornl.gov kernel: [1915824.957831] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880eb2948400 x1471904508186988/t0(0) o38->atlas2-MDT0000-lwp-OST0375@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3c6.ccs.ornl.gov kernel: [1915825.042028] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3c7.ccs.ornl.gov kernel: [1915833.576781] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880646dba000 x1471904510248788/t0(0) o38->atlas2-MDT0000-lwp-OST0376@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3c7.ccs.ornl.gov kernel: [1915833.663385] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3d3.ccs.ornl.gov kernel: [1915839.542336] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88059d715c00 x1471904510125820/t0(0) o38->atlas2-MDT0000-lwp-OST025a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3d3.ccs.ornl.gov kernel: [1915839.629024] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3e4.ccs.ornl.gov kernel: [1915846.961186] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8806e30ce400 x1471904508538136/t0(0) o38->atlas2-MDT0000-lwp-OST0383@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4a3.ccs.ornl.gov kernel: [1915848.684291] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8801e0172400 x1471904514689004/t0(0) o38->atlas2-MDT0000-lwp-OST03aa@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4a3.ccs.ornl.gov kernel: [1915848.771705] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3f3.ccs.ornl.gov kernel: [1915846.309307] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4c6.ccs.ornl.gov kernel: [1915833.632142] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880c4c74a800 x1471904517937336/t0(0) o38->atlas2-MDT0000-lwp-OST03bd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4c6.ccs.ornl.gov kernel: [1915833.722905] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3c8.ccs.ornl.gov kernel: [1915837.769833] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880e0659c400 x1471904507928156/t0(0) o38->atlas2-MDT0000-lwp-OST0377@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3c8.ccs.ornl.gov kernel: [1915837.858360] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4a7.ccs.ornl.gov kernel: [1915844.385134] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880629099c00 x1471904510936132/t0(0) o38->atlas2-MDT0000-lwp-OST03ae@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3e2.ccs.ornl.gov kernel: [1915832.118798] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8806cb1c6400 x1471904513179304/t0(0) o38->atlas2-MDT0000-lwp-OST0381@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3e2.ccs.ornl.gov kernel: [1915832.208085] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4a1.ccs.ornl.gov kernel: [1915830.676329] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880ffde49c00 x1471904510345916/t0(0) o38->atlas2-MDT0000-lwp-OST03a8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4e6.ccs.ornl.gov kernel: [1915848.486809] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880bf9a7a400 x1471904508408232/t0(0) o38->atlas2-MDT0000-lwp-OST03cd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3h6.ccs.ornl.gov kernel: [1915833.114192] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880dbbba7400 x1471904512216636/t0(0) o38->atlas2-MDT0000-lwp-OST039d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3h6.ccs.ornl.gov kernel: [1915833.202961] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3d2.ccs.ornl.gov kernel: [1915834.458593] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880fedb78400 x1471904511214760/t0(0) o38->atlas2-MDT0000-lwp-OST0379@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3d2.ccs.ornl.gov kernel: [1915834.545343] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3i6.ccs.ornl.gov kernel: [1915831.608115] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8805607a1800 x1471904509634108/t0(0) o38->atlas2-MDT0000-lwp-OST0285@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3i6.ccs.ornl.gov kernel: [1915831.695260] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3e4.ccs.ornl.gov kernel: [1915847.054534] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3c4.ccs.ornl.gov kernel: [1915828.040644] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880490186c00 x1471904511833024/t0(0) o38->atlas2-MDT0000-lwp-OST01c3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3g3.ccs.ornl.gov kernel: [1914430.144293] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8804d6558800 x1471904516976872/t0(0) o38->atlas2-MDT0000-lwp-OST0392@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3g3.ccs.ornl.gov kernel: [1914430.229281] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4a7.ccs.ornl.gov kernel: [1915844.474026] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3i3.ccs.ornl.gov kernel: [1915828.236499] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88102431ec00 x1471904518169384/t0(0) o38->atlas2-MDT0000-lwp-OST03a2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3i3.ccs.ornl.gov kernel: [1915828.323142] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3a7.ccs.ornl.gov kernel: [1915830.893130] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8805f3c34c00 x1471904518655740/t0(0) o38->atlas2-MDT0000-lwp-OST0366@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3a7.ccs.ornl.gov kernel: [1915830.982064] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4d1.ccs.ornl.gov kernel: [1915828.789585] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880bc4229c00 x1471904516811820/t0(0) o38->atlas2-MDT0000-lwp-OST03c0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4d1.ccs.ornl.gov kernel: [1915828.882860] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4a1.ccs.ornl.gov kernel: [1915830.767054] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4e6.ccs.ornl.gov kernel: [1915848.574467] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3g6.ccs.ornl.gov kernel: [1914441.145288] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880e1159d400 x1471904523473936/t0(0) o38->atlas2-MDT0000-lwp-OST0305@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4h3.ccs.ornl.gov kernel: [1915845.466409] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8808b49e1c00 x1471904509650956/t0(0) o38->atlas2-MDT0000-lwp-OST03e2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3c3.ccs.ornl.gov kernel: [1915825.469307] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88073022b400 x1471904511051556/t0(0) o38->atlas2-MDT0000-lwp-OST0372@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3c3.ccs.ornl.gov kernel: [1915825.554419] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3d7.ccs.ornl.gov kernel: [1915833.649950] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880d48bab400 x1471904512149560/t0(0) o38->atlas2-MDT0000-lwp-OST037e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4i1.ccs.ornl.gov kernel: [1915832.899176] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88032e635000 x1471904513319840/t0(0) o38->atlas2-MDT0000-lwp-OST0358@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3e6.ccs.ornl.gov kernel: [1915830.351477] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880fe8d0b400 x1471904520534736/t0(0) o38->atlas2-MDT0000-lwp-OST01d5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3c4.ccs.ornl.gov kernel: [1915828.127246] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4a8.ccs.ornl.gov kernel: [1915838.829868] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880e2d032800 x1471904507750876/t0(0) o38->atlas2-MDT0000-lwp-OST03af@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4a8.ccs.ornl.gov kernel: [1915838.916496] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3f1.ccs.ornl.gov kernel: [1915836.622757] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88055a52d000 x1471904508988540/t0(0) o38->atlas2-MDT0000-lwp-OST0388@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4b4.ccs.ornl.gov kernel: [1915830.896105] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88087043dc00 x1471904528872128/t0(0) o38->atlas2-MDT0000-lwp-OST03b3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4b4.ccs.ornl.gov kernel: [1915830.982512] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3b7.ccs.ornl.gov kernel: [1915837.682086] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88026bc45800 x1471904519392752/t0(0) o38->atlas2-MDT0000-lwp-OST036e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3b7.ccs.ornl.gov kernel: [1915837.769042] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4f8.ccs.ornl.gov kernel: [1915832.039699] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff8807962f5c00 x1471904511806136/t0(0) o38->atlas2-MDT0000-lwp-OST0347@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4f8.ccs.ornl.gov kernel: [1915832.132649] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3b1.ccs.ornl.gov kernel: [1915839.453445] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880628989c00 x1471904516110856/t0(0) o38->atlas2-MDT0000-lwp-OST0368@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3i1.ccs.ornl.gov kernel: [1915832.957110] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880283e9b400 x1471904519755516/t0(0) o38->atlas2-MDT0000-lwp-OST03a0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3i1.ccs.ornl.gov kernel: [1915833.041416] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3g6.ccs.ornl.gov kernel: [1914441.233395] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3e6.ccs.ornl.gov kernel: [1915830.438815] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3f1.ccs.ornl.gov kernel: [1915836.707382] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4h3.ccs.ornl.gov kernel: [1915845.552664] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4d8.ccs.ornl.gov kernel: [1915845.872448] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880839db9800 x1471904516223104/t0(0) o38->atlas2-MDT0000-lwp-OST03c7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss4d8.ccs.ornl.gov kernel: [1915845.957118] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3d7.ccs.ornl.gov kernel: [1915833.736236] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss4i1.ccs.ornl.gov kernel: [1915832.986263] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3b1.ccs.ornl.gov kernel: [1915839.539481] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3b8.ccs.ornl.gov kernel: [1915846.999997] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880baf01b400 x1471904516406888/t0(0) o38->atlas2-MDT0000-lwp-OST036f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3b8.ccs.ornl.gov kernel: [1915847.085871] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3d5.ccs.ornl.gov kernel: [1915836.354380] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880543c21800 x1471904509642140/t0(0) o38->atlas2-MDT0000-lwp-OST037c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3d5.ccs.ornl.gov kernel: [1915836.441822] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3b4.ccs.ornl.gov kernel: [1915844.523253] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff880e0fc91000 x1471904510548664/t0(0) o38->atlas2-MDT0000-lwp-OST02db@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3b4.ccs.ornl.gov kernel: [1915844.608514] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:04:37 atlas-oss3i2.ccs.ornl.gov kernel: [1915839.066653] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630797/real 1405630797] req@ffff88035c725800 x1471904514016364/t0(0) o38->atlas2-MDT0000-lwp-OST0281@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:04:37 atlas-oss3i2.ccs.ornl.gov kernel: [1915839.153175] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:05:23 atlas-oss3i5.ccs.ornl.gov kernel: [702186.898836] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630843/real 1405630843] req@ffff8804ba8d5800 x1473175721933444/t0(0) o38->atlas2-MDT0000-lwp-OST0314@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631123 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:05:23 atlas-oss3i5.ccs.ornl.gov kernel: [702186.986137] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:05:23 atlas-oss3a8.ccs.ornl.gov kernel: [702187.831486] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630843/real 1405630843] req@ffff88078d3b6800 x1473175720195024/t0(0) o38->atlas2-MDT0000-lwp-OST02d7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631123 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:05:23 atlas-oss3a8.ccs.ornl.gov kernel: [702187.918969] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:05:24 atlas-oss3d4.ccs.ornl.gov kernel: [702187.381487] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405630843/real 1405630843] req@ffff880769c44c00 x1473175723812824/t0(0) o38->atlas2-MDT0000-lwp-OST037b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631123 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:05:24 atlas-oss3d4.ccs.ornl.gov kernel: [702187.467547] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:53 atlas-oss3b5.ccs.ornl.gov kernel: [1915982.212253] LNetError: 25848:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:53 atlas-oss3b5.ccs.ornl.gov kernel: [1915982.239140] LNetError: 25848:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:53 atlas-oss3c2.ccs.ornl.gov kernel: [1915971.276821] LNetError: 25752:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:53 atlas-oss3c2.ccs.ornl.gov kernel: [1915971.303050] LNetError: 25752:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:53 atlas-oss3c3.ccs.ornl.gov kernel: [1915961.917317] LNetError: 22114:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:53 atlas-oss3c3.ccs.ornl.gov kernel: [1915961.945126] LNetError: 22114:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:53 atlas-oss3c1.ccs.ornl.gov kernel: [1915971.355680] LNetError: 23570:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:53 atlas-oss3c1.ccs.ornl.gov kernel: [1915971.387135] LNetError: 23570:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:53 atlas-oss3b6.ccs.ornl.gov kernel: [1915967.470696] LNetError: 25850:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:53 atlas-oss3b6.ccs.ornl.gov kernel: [1915967.498574] LNetError: 25850:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:53 atlas-oss3c4.ccs.ornl.gov kernel: [1915964.531820] LNetError: 25842:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:53 atlas-oss3c4.ccs.ornl.gov kernel: [1915964.557605] LNetError: 25842:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:53 atlas-oss3e2.ccs.ornl.gov kernel: [1915968.642174] LNetError: 27089:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:53 atlas-oss3e2.ccs.ornl.gov kernel: [1915968.666673] LNetError: 27089:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:53 atlas-oss3e3.ccs.ornl.gov kernel: [1915965.936955] LNetError: 22957:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:53 atlas-oss3e3.ccs.ornl.gov kernel: [1915965.960363] LNetError: 22957:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:53 atlas-oss3d7.ccs.ornl.gov kernel: [1915970.120347] LNetError: 22113:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:53 atlas-oss3d7.ccs.ornl.gov kernel: [1915970.147110] LNetError: 22113:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:53 atlas-oss3b7.ccs.ornl.gov kernel: [1915974.144737] LNetError: 25929:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:53 atlas-oss3b7.ccs.ornl.gov kernel: [1915974.169847] LNetError: 25929:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:53 atlas-oss3d8.ccs.ornl.gov kernel: [1915975.814468] LNetError: 24422:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:53 atlas-oss3d8.ccs.ornl.gov kernel: [1915975.839823] LNetError: 24422:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:53 atlas-oss3b8.ccs.ornl.gov kernel: [1915983.420352] LNetError: 22112:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:53 atlas-oss3b8.ccs.ornl.gov kernel: [1915983.447325] LNetError: 22112:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:53 atlas-oss3e1.ccs.ornl.gov kernel: [1915972.010022] LNetError: 23498:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:53 atlas-oss3e1.ccs.ornl.gov kernel: [1915972.037622] LNetError: 23498:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3d6.ccs.ornl.gov kernel: [1915983.656077] LNetError: 25502:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3d6.ccs.ornl.gov kernel: [1915983.683273] LNetError: 25502:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3d5.ccs.ornl.gov kernel: [1915972.757038] LNetError: 22073:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3d5.ccs.ornl.gov kernel: [1915972.782553] LNetError: 22073:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3d1.ccs.ornl.gov kernel: [1915961.873089] LNetError: 25555:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3d1.ccs.ornl.gov kernel: [1915961.896319] LNetError: 25555:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3d1.ccs.ornl.gov kernel: [1915961.927201] LNetError: 25555:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3d1.ccs.ornl.gov kernel: [1915961.967383] LNetError: 25555:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3e4.ccs.ornl.gov kernel: [1915983.515910] LNetError: 21991:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3e4.ccs.ornl.gov kernel: [1915983.546629] LNetError: 21991:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4a1.ccs.ornl.gov kernel: [1915967.223511] LNetError: 25876:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4a1.ccs.ornl.gov kernel: [1915967.247879] LNetError: 25876:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4c2.ccs.ornl.gov kernel: [1915973.203391] LNetError: 25817:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4c2.ccs.ornl.gov kernel: [1915973.232135] LNetError: 25817:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3h7.ccs.ornl.gov kernel: [1915975.991171] LNetError: 24976:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3h7.ccs.ornl.gov kernel: [1915976.018120] LNetError: 24976:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3b1.ccs.ornl.gov kernel: [1915975.868413] LNetError: 22206:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3b1.ccs.ornl.gov kernel: [1915975.891622] LNetError: 22206:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3b1.ccs.ornl.gov kernel: [1915975.930516] LNetError: 22206:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3b1.ccs.ornl.gov kernel: [1915975.970925] LNetError: 22206:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4b6.ccs.ornl.gov kernel: [1915966.888608] LNetError: 25984:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4b6.ccs.ornl.gov kernel: [1915966.919423] LNetError: 25984:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4d6.ccs.ornl.gov kernel: [1915969.377961] LNetError: 25440:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4d6.ccs.ornl.gov kernel: [1915969.401066] LNetError: 25440:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3c5.ccs.ornl.gov kernel: [1915975.831123] LNetError: 25691:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3c5.ccs.ornl.gov kernel: [1915975.856758] LNetError: 25691:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3c5.ccs.ornl.gov kernel: [1915975.887670] LNetError: 25691:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3c5.ccs.ornl.gov kernel: [1915975.936331] LNetError: 25691:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3e7.ccs.ornl.gov kernel: [1915971.617570] LNetError: 25771:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3e7.ccs.ornl.gov kernel: [1915971.640422] LNetError: 25771:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3e7.ccs.ornl.gov kernel: [1915971.670768] LNetError: 25771:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3e7.ccs.ornl.gov kernel: [1915971.710988] LNetError: 25771:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4c3.ccs.ornl.gov kernel: [1915966.069866] LNetError: 24855:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4b8.ccs.ornl.gov kernel: [1915971.147000] LNetError: 25686:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4c3.ccs.ornl.gov kernel: [1915966.094091] LNetError: 24855:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4b8.ccs.ornl.gov kernel: [1915971.171624] LNetError: 25686:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4b7.ccs.ornl.gov kernel: [1915963.355739] LNetError: 23047:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4b7.ccs.ornl.gov kernel: [1915963.378761] LNetError: 23047:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4c5.ccs.ornl.gov kernel: [1915965.033295] LNetError: 26551:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4c5.ccs.ornl.gov kernel: [1915965.063192] LNetError: 26551:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4e5.ccs.ornl.gov kernel: [1915970.383394] LNetError: 24539:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4e5.ccs.ornl.gov kernel: [1915970.407555] LNetError: 24539:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4b5.ccs.ornl.gov kernel: [1915986.857721] LNetError: 25619:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4b5.ccs.ornl.gov kernel: [1915986.885920] LNetError: 25619:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4c4.ccs.ornl.gov kernel: [1915981.153803] LNetError: 25681:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4c4.ccs.ornl.gov kernel: [1915981.184686] LNetError: 25681:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4e3.ccs.ornl.gov kernel: [1915975.110859] LNetError: 27250:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4e3.ccs.ornl.gov kernel: [1915975.136744] LNetError: 27250:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3b4.ccs.ornl.gov kernel: [1915980.806366] LNetError: 26810:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4b2.ccs.ornl.gov kernel: [1915992.098062] LNetError: 27356:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3b4.ccs.ornl.gov kernel: [1915980.829685] LNetError: 26810:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3b4.ccs.ornl.gov kernel: [1915980.860212] LNetError: 26810:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3b4.ccs.ornl.gov kernel: [1915980.909134] LNetError: 26810:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3a7.ccs.ornl.gov kernel: [1915967.361367] LNetError: 21971:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4b2.ccs.ornl.gov kernel: [1915992.129012] LNetError: 27356:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3a7.ccs.ornl.gov kernel: [1915967.392755] LNetError: 21971:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3a7.ccs.ornl.gov kernel: [1915967.423411] LNetError: 21971:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3a7.ccs.ornl.gov kernel: [1915967.463752] LNetError: 21971:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3a2.ccs.ornl.gov kernel: [1915968.761859] LNetError: 25515:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3a2.ccs.ornl.gov kernel: [1915968.786060] LNetError: 25515:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3a2.ccs.ornl.gov kernel: [1915968.816911] LNetError: 25515:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3a2.ccs.ornl.gov kernel: [1915968.865479] LNetError: 25515:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4c1.ccs.ornl.gov kernel: [1915974.793897] LNetError: 22025:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4d7.ccs.ornl.gov kernel: [1915969.994986] LNetError: 22169:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4c1.ccs.ornl.gov kernel: [1915974.820050] LNetError: 22025:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4d7.ccs.ornl.gov kernel: [1915970.021191] LNetError: 22169:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4e4.ccs.ornl.gov kernel: [1915977.993395] LNetError: 22083:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 17 seconds Jul 17 17:06:54 atlas-oss4e4.ccs.ornl.gov kernel: [1915978.018663] LNetError: 22083:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (117): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4c7.ccs.ornl.gov kernel: [1915969.173571] LNetError: 25503:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3c6.ccs.ornl.gov kernel: [1915961.486850] LNetError: 24430:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4c7.ccs.ornl.gov kernel: [1915969.197760] LNetError: 25503:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4d5.ccs.ornl.gov kernel: [1915981.806400] LNetError: 25983:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3c6.ccs.ornl.gov kernel: [1915961.511398] LNetError: 24430:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3c6.ccs.ornl.gov kernel: [1915961.542123] LNetError: 24430:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3c6.ccs.ornl.gov kernel: [1915961.591318] LNetError: 24430:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4d5.ccs.ornl.gov kernel: [1915981.837849] LNetError: 25983:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3b2.ccs.ornl.gov kernel: [1915969.228388] LNetError: 25169:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3b2.ccs.ornl.gov kernel: [1915969.258507] LNetError: 25169:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3b2.ccs.ornl.gov kernel: [1915969.289064] LNetError: 25169:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3b2.ccs.ornl.gov kernel: [1915969.329114] LNetError: 25169:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3b5.ccs.ornl.gov kernel: [1915982.279915] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631213] req@ffff8806d9c73000 x1471904514717116/t0(0) o38->atlas2-MDT0000-lwp-OST036c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3b5.ccs.ornl.gov kernel: [1915982.379228] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3a4.ccs.ornl.gov kernel: [1915959.223840] LNetError: 22072:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3a4.ccs.ornl.gov kernel: [1915959.249740] LNetError: 22072:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3a4.ccs.ornl.gov kernel: [1915959.280541] LNetError: 22072:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3a4.ccs.ornl.gov kernel: [1915959.320509] LNetError: 22072:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4e2.ccs.ornl.gov kernel: [1915983.111189] LNetError: 22062:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4e2.ccs.ornl.gov kernel: [1915983.140561] LNetError: 22062:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3a3.ccs.ornl.gov kernel: [1915966.464476] LNetError: 27104:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3a3.ccs.ornl.gov kernel: [1915966.495484] LNetError: 27104:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3a3.ccs.ornl.gov kernel: [1915966.526252] LNetError: 27104:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3a3.ccs.ornl.gov kernel: [1915966.574969] LNetError: 27104:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3c7.ccs.ornl.gov kernel: [1915970.120232] LNetError: 25644:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3c7.ccs.ornl.gov kernel: [1915970.144429] LNetError: 25644:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3c7.ccs.ornl.gov kernel: [1915970.174867] LNetError: 25644:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3c7.ccs.ornl.gov kernel: [1915970.215056] LNetError: 25644:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3a6.ccs.ornl.gov kernel: [1915971.865746] LNetError: 25668:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3a6.ccs.ornl.gov kernel: [1915971.896434] LNetError: 25668:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3a6.ccs.ornl.gov kernel: [1915971.927232] LNetError: 25668:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3a6.ccs.ornl.gov kernel: [1915971.975743] LNetError: 25668:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4g6.ccs.ornl.gov kernel: [1915978.994261] LNetError: 24543:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4g6.ccs.ornl.gov kernel: [1915979.017570] LNetError: 24543:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3c8.ccs.ornl.gov kernel: [1915974.293468] LNetError: 25920:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3c8.ccs.ornl.gov kernel: [1915974.319851] LNetError: 25920:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3c8.ccs.ornl.gov kernel: [1915974.350350] LNetError: 25920:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3c8.ccs.ornl.gov kernel: [1915974.390639] LNetError: 25920:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4i7.ccs.ornl.gov kernel: [1915982.596826] LNetError: 25044:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 17 seconds Jul 17 17:06:54 atlas-oss4i7.ccs.ornl.gov kernel: [1915982.620803] LNetError: 25044:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (117): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3d3.ccs.ornl.gov kernel: [1915976.068285] LNetError: 25435:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3d3.ccs.ornl.gov kernel: [1915976.099993] LNetError: 25435:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3d3.ccs.ornl.gov kernel: [1915976.130927] LNetError: 25435:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3d3.ccs.ornl.gov kernel: [1915976.171295] LNetError: 25435:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4d8.ccs.ornl.gov kernel: [1915982.367815] LNetError: 22007:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4d8.ccs.ornl.gov kernel: [1915982.397857] LNetError: 22007:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3d2.ccs.ornl.gov kernel: [1915970.970890] LNetError: 25867:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3d2.ccs.ornl.gov kernel: [1915970.996216] LNetError: 25867:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3d2.ccs.ornl.gov kernel: [1915971.026787] LNetError: 25867:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3d2.ccs.ornl.gov kernel: [1915971.066825] LNetError: 25867:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3f2.ccs.ornl.gov kernel: [1915964.378728] LNetError: 25605:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3f2.ccs.ornl.gov kernel: [1915964.409750] LNetError: 25605:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3f2.ccs.ornl.gov kernel: [1915964.440466] LNetError: 25605:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3f2.ccs.ornl.gov kernel: [1915964.480661] LNetError: 25605:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4e1.ccs.ornl.gov kernel: [1915967.173968] LNetError: 25895:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4e1.ccs.ornl.gov kernel: [1915967.204900] LNetError: 25895:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3g1.ccs.ornl.gov kernel: [1914559.214051] LNetError: 13726:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3f3.ccs.ornl.gov kernel: [1915982.782031] LNetError: 25904:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3g1.ccs.ornl.gov kernel: [1914559.239184] LNetError: 13726:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3g1.ccs.ornl.gov kernel: [1914559.278997] LNetError: 13726:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3g1.ccs.ornl.gov kernel: [1914559.319402] LNetError: 13726:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3f3.ccs.ornl.gov kernel: [1915982.810684] LNetError: 25904:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3f3.ccs.ornl.gov kernel: [1915982.841328] LNetError: 25904:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3f3.ccs.ornl.gov kernel: [1915982.881673] LNetError: 25904:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3e6.ccs.ornl.gov kernel: [1915966.824724] LNetError: 21991:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3a5.ccs.ornl.gov kernel: [1915962.778024] LNetError: 22121:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3e6.ccs.ornl.gov kernel: [1915966.849526] LNetError: 21991:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3e6.ccs.ornl.gov kernel: [1915966.880345] LNetError: 21991:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3e6.ccs.ornl.gov kernel: [1915966.928833] LNetError: 21991:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3a5.ccs.ornl.gov kernel: [1915962.806683] LNetError: 22121:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3a5.ccs.ornl.gov kernel: [1915962.837239] LNetError: 22121:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3a5.ccs.ornl.gov kernel: [1915962.877465] LNetError: 22121:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3c2.ccs.ornl.gov kernel: [1915971.343763] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631213] req@ffff880456989800 x1471904511648084/t0(0) o38->atlas2-MDT0000-lwp-OST0371@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3c2.ccs.ornl.gov kernel: [1915971.442710] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3g2.ccs.ornl.gov kernel: [1914565.890453] LNetError: 13833:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3g2.ccs.ornl.gov kernel: [1914565.915029] LNetError: 13833:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3g2.ccs.ornl.gov kernel: [1914565.945871] LNetError: 13833:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3g2.ccs.ornl.gov kernel: [1914565.994427] LNetError: 13833:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3c1.ccs.ornl.gov kernel: [1915971.427880] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631213] req@ffff8801bfe83400 x1471904511393124/t0(0) o38->atlas2-MDT0000-lwp-OST02e0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3c1.ccs.ornl.gov kernel: [1915971.518354] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3g7.ccs.ornl.gov kernel: [1914566.810642] LNetError: 13344:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3g7.ccs.ornl.gov kernel: [1914566.840021] LNetError: 13344:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3g7.ccs.ornl.gov kernel: [1914566.870857] LNetError: 13344:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3g7.ccs.ornl.gov kernel: [1914566.910935] LNetError: 13344:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3c3.ccs.ornl.gov kernel: [1915961.985789] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631213] req@ffff88056b774c00 x1471904511051592/t0(0) o38->atlas2-MDT0000-lwp-OST0372@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3c3.ccs.ornl.gov kernel: [1915962.084576] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3g5.ccs.ornl.gov kernel: [1914570.840835] LNetError: 13395:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3g5.ccs.ornl.gov kernel: [1914570.868354] LNetError: 13395:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3g5.ccs.ornl.gov kernel: [1914570.899067] LNetError: 13395:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3g5.ccs.ornl.gov kernel: [1914570.939347] LNetError: 13395:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3a1.ccs.ornl.gov kernel: [1915964.072825] LNetError: 22326:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3a1.ccs.ornl.gov kernel: [1915964.096846] LNetError: 22326:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3a1.ccs.ornl.gov kernel: [1915964.127236] LNetError: 22326:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3a1.ccs.ornl.gov kernel: [1915964.175957] LNetError: 22326:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3h6.ccs.ornl.gov kernel: [1915969.640703] LNetError: 25640:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3h6.ccs.ornl.gov kernel: [1915969.663701] LNetError: 25640:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3h6.ccs.ornl.gov kernel: [1915969.694668] LNetError: 25640:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3h6.ccs.ornl.gov kernel: [1915969.734811] LNetError: 25640:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3h2.ccs.ornl.gov kernel: [1915972.305123] LNetError: 20811:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3h2.ccs.ornl.gov kernel: [1915972.329967] LNetError: 20811:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3h2.ccs.ornl.gov kernel: [1915972.360845] LNetError: 20811:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3h2.ccs.ornl.gov kernel: [1915972.400818] LNetError: 20811:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3i3.ccs.ornl.gov kernel: [1915964.729601] LNetError: 22025:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3f1.ccs.ornl.gov kernel: [1915973.103363] LNetError: 24103:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3i3.ccs.ornl.gov kernel: [1915964.764228] LNetError: 22025:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3i3.ccs.ornl.gov kernel: [1915964.794894] LNetError: 22025:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3i3.ccs.ornl.gov kernel: [1915964.843648] LNetError: 22025:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3f1.ccs.ornl.gov kernel: [1915973.129509] LNetError: 24103:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3f1.ccs.ornl.gov kernel: [1915973.159926] LNetError: 24103:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3f1.ccs.ornl.gov kernel: [1915973.200271] LNetError: 24103:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3f4.ccs.ornl.gov kernel: [1915971.428883] LNetError: 22046:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3f4.ccs.ornl.gov kernel: [1915971.460405] LNetError: 22046:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3f4.ccs.ornl.gov kernel: [1915971.491253] LNetError: 22046:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3f4.ccs.ornl.gov kernel: [1915971.531410] LNetError: 22046:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4h5.ccs.ornl.gov kernel: [1915971.835397] LNetError: 25564:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4h5.ccs.ornl.gov kernel: [1915971.858362] LNetError: 25564:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3h3.ccs.ornl.gov kernel: [1915964.861043] LNetError: 26518:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3h3.ccs.ornl.gov kernel: [1915964.884330] LNetError: 26518:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3h3.ccs.ornl.gov kernel: [1915964.915119] LNetError: 26518:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3h3.ccs.ornl.gov kernel: [1915964.955111] LNetError: 26518:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3e5.ccs.ornl.gov kernel: [1915970.512889] LNetError: 25835:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3e5.ccs.ornl.gov kernel: [1915970.539948] LNetError: 25835:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3e5.ccs.ornl.gov kernel: [1915970.570628] LNetError: 25835:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3e5.ccs.ornl.gov kernel: [1915970.619402] LNetError: 25835:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4i3.ccs.ornl.gov kernel: [1915961.571226] LNetError: 25844:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4i3.ccs.ornl.gov kernel: [1915961.595374] LNetError: 25844:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4h2.ccs.ornl.gov kernel: [1915990.566007] LNetError: 25891:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 17 seconds Jul 17 17:06:54 atlas-oss4g7.ccs.ornl.gov kernel: [1915991.634330] LNetError: 25950:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3b6.ccs.ornl.gov kernel: [1915967.539522] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631213] req@ffff8804c6230c00 x1471904512275528/t0(0) o38->atlas2-MDT0000-lwp-OST036d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4h2.ccs.ornl.gov kernel: [1915990.591369] LNetError: 25891:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (117): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4g7.ccs.ornl.gov kernel: [1915991.665791] LNetError: 25950:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3b6.ccs.ornl.gov kernel: [1915967.638293] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3h5.ccs.ornl.gov kernel: [1915988.982901] LNetError: 24487:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3h5.ccs.ornl.gov kernel: [1915989.006124] LNetError: 24487:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3h5.ccs.ornl.gov kernel: [1915989.045213] LNetError: 24487:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3h5.ccs.ornl.gov kernel: [1915989.085394] LNetError: 24487:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3c4.ccs.ornl.gov kernel: [1915964.598279] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631213] req@ffff8804a8b79c00 x1471904511833068/t0(0) o38->atlas2-MDT0000-lwp-OST0373@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3c4.ccs.ornl.gov kernel: [1915964.697286] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3g6.ccs.ornl.gov kernel: [1914577.651709] LNetError: 13821:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3g6.ccs.ornl.gov kernel: [1914577.674982] LNetError: 13821:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3g6.ccs.ornl.gov kernel: [1914577.705844] LNetError: 13821:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3g6.ccs.ornl.gov kernel: [1914577.745939] LNetError: 13821:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3h8.ccs.ornl.gov kernel: [1915966.132610] LNetError: 22071:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3h8.ccs.ornl.gov kernel: [1915966.164111] LNetError: 22071:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3h8.ccs.ornl.gov kernel: [1915966.194831] LNetError: 22071:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3h8.ccs.ornl.gov kernel: [1915966.243295] LNetError: 22071:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3b3.ccs.ornl.gov kernel: [1915962.239022] LNetError: 25480:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4g5.ccs.ornl.gov kernel: [1915969.451918] LNetError: 25945:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 17 seconds Jul 17 17:06:54 atlas-oss4g5.ccs.ornl.gov kernel: [1915969.475373] LNetError: 25945:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (117): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3b3.ccs.ornl.gov kernel: [1915962.267346] LNetError: 25480:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3b3.ccs.ornl.gov kernel: [1915962.297989] LNetError: 25480:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3b3.ccs.ornl.gov kernel: [1915962.338117] LNetError: 25480:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3g3.ccs.ornl.gov kernel: [1914566.666984] LNetError: 13844:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3i2.ccs.ornl.gov kernel: [1915975.199440] LNetError: 25606:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3g3.ccs.ornl.gov kernel: [1914566.690126] LNetError: 13844:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3g3.ccs.ornl.gov kernel: [1914566.720971] LNetError: 13844:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3g3.ccs.ornl.gov kernel: [1914566.769557] LNetError: 13844:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3i2.ccs.ornl.gov kernel: [1915975.224311] LNetError: 25606:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3i2.ccs.ornl.gov kernel: [1915975.255144] LNetError: 25606:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3i2.ccs.ornl.gov kernel: [1915975.295325] LNetError: 25606:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3i7.ccs.ornl.gov kernel: [1915966.425400] LNetError: 24468:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3i7.ccs.ornl.gov kernel: [1915966.456795] LNetError: 24468:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3i7.ccs.ornl.gov kernel: [1915966.487583] LNetError: 24468:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3i7.ccs.ornl.gov kernel: [1915966.527771] LNetError: 24468:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4b1.ccs.ornl.gov kernel: [1915973.767746] LNetError: 25757:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4b1.ccs.ornl.gov kernel: [1915973.794371] LNetError: 25757:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4b1.ccs.ornl.gov kernel: [1915973.825235] LNetError: 25757:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4b1.ccs.ornl.gov kernel: [1915973.873811] LNetError: 25757:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3h4.ccs.ornl.gov kernel: [1915973.053599] LNetError: 25629:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4a7.ccs.ornl.gov kernel: [1915980.942405] LNetError: 25669:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3h4.ccs.ornl.gov kernel: [1915973.082111] LNetError: 25629:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3h4.ccs.ornl.gov kernel: [1915973.112761] LNetError: 25629:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3h4.ccs.ornl.gov kernel: [1915973.161382] LNetError: 25629:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4a7.ccs.ornl.gov kernel: [1915980.965979] LNetError: 25669:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4a7.ccs.ornl.gov kernel: [1915980.996786] LNetError: 25669:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4a7.ccs.ornl.gov kernel: [1915981.045327] LNetError: 25669:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3e2.ccs.ornl.gov kernel: [1915968.707474] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631213] req@ffff880167629800 x1471904513179336/t0(0) o38->atlas2-MDT0000-lwp-OST02f1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4i6.ccs.ornl.gov kernel: [1915982.733246] LNetError: 25614:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4i6.ccs.ornl.gov kernel: [1915982.757972] LNetError: 25614:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3e3.ccs.ornl.gov kernel: [1915966.001028] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631213] req@ffff88085f698400 x1471904510468604/t0(0) o38->atlas2-MDT0000-lwp-OST0382@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3i6.ccs.ornl.gov kernel: [1915968.140284] LNetError: 25596:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3d7.ccs.ornl.gov kernel: [1915970.187752] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631213] req@ffff880ef54ac800 x1471904512149596/t0(0) o38->atlas2-MDT0000-lwp-OST037e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3i6.ccs.ornl.gov kernel: [1915968.166088] LNetError: 25596:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3i6.ccs.ornl.gov kernel: [1915968.196614] LNetError: 25596:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3i6.ccs.ornl.gov kernel: [1915968.236903] LNetError: 25596:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3g8.ccs.ornl.gov kernel: [1914567.322478] LNetError: 13335:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4i1.ccs.ornl.gov kernel: [1915969.465526] LNetError: 25662:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3g8.ccs.ornl.gov kernel: [1914567.353340] LNetError: 13335:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3g8.ccs.ornl.gov kernel: [1914567.384171] LNetError: 13335:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3g8.ccs.ornl.gov kernel: [1914567.424357] LNetError: 13335:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4i1.ccs.ornl.gov kernel: [1915969.496784] LNetError: 25662:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4a3.ccs.ornl.gov kernel: [1915985.249851] LNetError: 28086:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4a3.ccs.ornl.gov kernel: [1915985.273952] LNetError: 28086:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4a3.ccs.ornl.gov kernel: [1915985.304571] LNetError: 28086:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4a3.ccs.ornl.gov kernel: [1915985.345033] LNetError: 28086:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3b7.ccs.ornl.gov kernel: [1915974.210438] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631213] req@ffff8807672c1000 x1471904519392788/t0(0) o38->atlas2-MDT0000-lwp-OST036e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3b7.ccs.ornl.gov kernel: [1915974.309549] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4a2.ccs.ornl.gov kernel: [1915983.917315] LNetError: 25673:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3e8.ccs.ornl.gov kernel: [1915969.406034] LNetError: 25679:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3e8.ccs.ornl.gov kernel: [1915969.431417] LNetError: 25679:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3e8.ccs.ornl.gov kernel: [1915969.462383] LNetError: 25679:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3e8.ccs.ornl.gov kernel: [1915969.510990] LNetError: 25679:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4a2.ccs.ornl.gov kernel: [1915983.940231] LNetError: 25673:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4a2.ccs.ornl.gov kernel: [1915983.979099] LNetError: 25673:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4a2.ccs.ornl.gov kernel: [1915984.019300] LNetError: 25673:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4b3.ccs.ornl.gov kernel: [1915963.580687] LNetError: 22190:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4b3.ccs.ornl.gov kernel: [1915963.610973] LNetError: 22190:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4b3.ccs.ornl.gov kernel: [1915963.641645] LNetError: 22190:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (117): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4b3.ccs.ornl.gov kernel: [1915963.681907] LNetError: 22190:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4b4.ccs.ornl.gov kernel: [1915967.392171] LNetError: 25907:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3d8.ccs.ornl.gov kernel: [1915975.880990] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631213] req@ffff880fa86b0400 x1471904508710816/t0(0) o38->atlas2-MDT0000-lwp-OST02ef@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4b4.ccs.ornl.gov kernel: [1915967.423670] LNetError: 25907:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4b4.ccs.ornl.gov kernel: [1915967.454248] LNetError: 25907:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4b4.ccs.ornl.gov kernel: [1915967.494378] LNetError: 25907:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4a4.ccs.ornl.gov kernel: [1915970.835344] LNetError: 27243:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4a4.ccs.ornl.gov kernel: [1915970.862859] LNetError: 27243:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4a4.ccs.ornl.gov kernel: [1915970.893376] LNetError: 27243:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4a4.ccs.ornl.gov kernel: [1915970.942138] LNetError: 27243:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3i1.ccs.ornl.gov kernel: [1915969.430654] LNetError: 22021:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3h1.ccs.ornl.gov kernel: [1915957.990662] LNetError: 22100:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3i1.ccs.ornl.gov kernel: [1915969.462228] LNetError: 22021:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3i1.ccs.ornl.gov kernel: [1915969.493225] LNetError: 22021:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3i1.ccs.ornl.gov kernel: [1915969.533397] LNetError: 22021:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3i8.ccs.ornl.gov kernel: [1915983.188143] LNetError: 22063:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3i8.ccs.ornl.gov kernel: [1915983.211226] LNetError: 22063:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3i8.ccs.ornl.gov kernel: [1915983.241707] LNetError: 22063:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3i8.ccs.ornl.gov kernel: [1915983.290469] LNetError: 22063:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3h1.ccs.ornl.gov kernel: [1915958.020523] LNetError: 22100:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3h1.ccs.ornl.gov kernel: [1915958.051360] LNetError: 22100:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3h1.ccs.ornl.gov kernel: [1915958.091510] LNetError: 22100:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3i4.ccs.ornl.gov kernel: [1915975.154676] LNetError: 27159:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3i4.ccs.ornl.gov kernel: [1915975.182409] LNetError: 27159:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3i4.ccs.ornl.gov kernel: [1915975.213392] LNetError: 27159:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3i4.ccs.ornl.gov kernel: [1915975.262051] LNetError: 27159:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4a5.ccs.ornl.gov kernel: [1915988.255233] LNetError: 22017:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4a5.ccs.ornl.gov kernel: [1915988.279409] LNetError: 22017:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4a5.ccs.ornl.gov kernel: [1915988.310116] LNetError: 22017:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4a5.ccs.ornl.gov kernel: [1915988.358880] LNetError: 22017:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4e7.ccs.ornl.gov kernel: [1915966.694086] LNetError: 25983:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4e7.ccs.ornl.gov kernel: [1915966.721708] LNetError: 25983:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4e7.ccs.ornl.gov kernel: [1915966.752377] LNetError: 25983:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4e7.ccs.ornl.gov kernel: [1915966.801278] LNetError: 25983:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3b8.ccs.ornl.gov kernel: [1915983.488039] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631213] req@ffff880bc9162c00 x1471904516406924/t0(0) o38->atlas2-MDT0000-lwp-OST036f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3b8.ccs.ornl.gov kernel: [1915983.586991] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4c6.ccs.ornl.gov kernel: [1915970.207668] LNetError: 22077:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4a8.ccs.ornl.gov kernel: [1915975.335717] LNetError: 25846:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4c6.ccs.ornl.gov kernel: [1915970.234190] LNetError: 22077:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4c6.ccs.ornl.gov kernel: [1915970.264757] LNetError: 22077:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4c6.ccs.ornl.gov kernel: [1915970.305157] LNetError: 22077:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3e1.ccs.ornl.gov kernel: [1915972.078399] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631213] req@ffff8801c64be000 x1471904509694496/t0(0) o38->atlas2-MDT0000-lwp-OST02f0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3e1.ccs.ornl.gov kernel: [1915972.177313] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4a8.ccs.ornl.gov kernel: [1915975.367369] LNetError: 25846:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4a8.ccs.ornl.gov kernel: [1915975.407586] LNetError: 25846:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4a8.ccs.ornl.gov kernel: [1915975.447924] LNetError: 25846:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4c8.ccs.ornl.gov kernel: [1915961.162111] LNetError: 21940:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4c8.ccs.ornl.gov kernel: [1915961.189180] LNetError: 21940:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4c8.ccs.ornl.gov kernel: [1915961.219535] LNetError: 21940:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4c8.ccs.ornl.gov kernel: [1915961.259556] LNetError: 21940:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4d3.ccs.ornl.gov kernel: [1915965.763730] LNetError: 22220:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4d3.ccs.ornl.gov kernel: [1915965.788048] LNetError: 22220:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4d3.ccs.ornl.gov kernel: [1915965.819145] LNetError: 22220:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4d3.ccs.ornl.gov kernel: [1915965.867684] LNetError: 22220:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4d4.ccs.ornl.gov kernel: [1915989.636925] LNetError: 27274:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4d4.ccs.ornl.gov kernel: [1915989.661512] LNetError: 27274:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4d4.ccs.ornl.gov kernel: [1915989.692017] LNetError: 27274:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4d4.ccs.ornl.gov kernel: [1915989.740772] LNetError: 27274:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4d1.ccs.ornl.gov kernel: [1915965.326815] LNetError: 22143:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4d1.ccs.ornl.gov kernel: [1915965.354235] LNetError: 22143:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4d1.ccs.ornl.gov kernel: [1915965.384935] LNetError: 22143:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4d1.ccs.ornl.gov kernel: [1915965.425139] LNetError: 22143:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3d6.ccs.ornl.gov kernel: [1915983.724363] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880f4b1d3800 x1471904511183532/t0(0) o38->atlas2-MDT0000-lwp-OST037d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3d6.ccs.ornl.gov kernel: [1915983.823178] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3d5.ccs.ornl.gov kernel: [1915972.823431] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880d48dea000 x1471904509642176/t0(0) o38->atlas2-MDT0000-lwp-OST037c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3d5.ccs.ornl.gov kernel: [1915972.922467] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3d1.ccs.ornl.gov kernel: [1915962.007583] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880711655800 x1471904515778460/t0(0) o38->atlas2-MDT0000-lwp-OST02e8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3d1.ccs.ornl.gov kernel: [1915962.106034] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4a1.ccs.ornl.gov kernel: [1915967.288849] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880a18140400 x1471904510345952/t0(0) o38->atlas2-MDT0000-lwp-OST03a8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3e4.ccs.ornl.gov kernel: [1915983.587582] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8805a8efc000 x1471904508538172/t0(0) o38->atlas2-MDT0000-lwp-OST0383@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4a6.ccs.ornl.gov kernel: [1915971.365778] LNetError: 25484:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4a6.ccs.ornl.gov kernel: [1915971.397029] LNetError: 25484:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4a6.ccs.ornl.gov kernel: [1915971.427758] LNetError: 25484:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4a6.ccs.ornl.gov kernel: [1915971.476325] LNetError: 25484:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4e6.ccs.ornl.gov kernel: [1915985.060389] LNetError: 25596:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4e6.ccs.ornl.gov kernel: [1915985.086326] LNetError: 25596:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4e6.ccs.ornl.gov kernel: [1915985.117310] LNetError: 25596:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4e6.ccs.ornl.gov kernel: [1915985.165978] LNetError: 25596:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4f1.ccs.ornl.gov kernel: [1915973.478033] LNetError: 25847:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4d2.ccs.ornl.gov kernel: [1915974.404720] LNetError: 27324:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 17 seconds Jul 17 17:06:54 atlas-oss4f1.ccs.ornl.gov kernel: [1915973.506960] LNetError: 25847:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4f1.ccs.ornl.gov kernel: [1915973.537755] LNetError: 25847:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4f1.ccs.ornl.gov kernel: [1915973.586459] LNetError: 25847:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4d2.ccs.ornl.gov kernel: [1915974.436218] LNetError: 27324:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4d2.ccs.ornl.gov kernel: [1915974.466968] LNetError: 27324:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (117): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4d2.ccs.ornl.gov kernel: [1915974.507102] LNetError: 27324:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4c2.ccs.ornl.gov kernel: [1915973.273655] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880fb0f13800 x1471904508570816/t0(0) o38->atlas2-MDT0000-lwp-OST03b9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3h7.ccs.ornl.gov kernel: [1915976.058635] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880725463400 x1471904510333092/t0(0) o38->atlas2-MDT0000-lwp-OST039e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4c2.ccs.ornl.gov kernel: [1915973.371989] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3h7.ccs.ornl.gov kernel: [1915976.157379] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3b1.ccs.ornl.gov kernel: [1915976.001928] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880358531800 x1471904516110896/t0(0) o38->atlas2-MDT0000-lwp-OST02d8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3e2.ccs.ornl.gov kernel: [1915968.806663] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3e3.ccs.ornl.gov kernel: [1915966.100052] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4b6.ccs.ornl.gov kernel: [1915966.960214] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88081dac7800 x1471904511213716/t0(0) o38->atlas2-MDT0000-lwp-OST03b5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4b6.ccs.ornl.gov kernel: [1915967.059292] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3e7.ccs.ornl.gov kernel: [1915971.750606] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880a5b420000 x1471904508195360/t0(0) o38->atlas2-MDT0000-lwp-OST0386@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3e7.ccs.ornl.gov kernel: [1915971.840862] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4f8.ccs.ornl.gov kernel: [1915968.565101] LNetError: 22106:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4f8.ccs.ornl.gov kernel: [1915968.594214] LNetError: 22106:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4f8.ccs.ornl.gov kernel: [1915968.624855] LNetError: 22106:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4f8.ccs.ornl.gov kernel: [1915968.673493] LNetError: 22106:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4f5.ccs.ornl.gov kernel: [1915967.239129] LNetError: 22035:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4f5.ccs.ornl.gov kernel: [1915967.263565] LNetError: 22035:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4f5.ccs.ornl.gov kernel: [1915967.294159] LNetError: 22035:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4f5.ccs.ornl.gov kernel: [1915967.334447] LNetError: 22035:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3d8.ccs.ornl.gov kernel: [1915975.979735] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4d6.ccs.ornl.gov kernel: [1915969.441520] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88073dea4400 x1471904519610564/t0(0) o38->atlas2-MDT0000-lwp-OST0335@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3c5.ccs.ornl.gov kernel: [1915975.967342] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88090b25fc00 x1471904512466272/t0(0) o38->atlas2-MDT0000-lwp-OST0374@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4e8.ccs.ornl.gov kernel: [1915982.275817] LNetError: 24486:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4e8.ccs.ornl.gov kernel: [1915982.302075] LNetError: 24486:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4e8.ccs.ornl.gov kernel: [1915982.332598] LNetError: 24486:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4e8.ccs.ornl.gov kernel: [1915982.381296] LNetError: 24486:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4g2.ccs.ornl.gov kernel: [1915980.088226] LNetError: 22025:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 17 seconds Jul 17 17:06:54 atlas-oss3d7.ccs.ornl.gov kernel: [1915970.287551] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4g2.ccs.ornl.gov kernel: [1915980.111640] LNetError: 22025:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4g2.ccs.ornl.gov kernel: [1915980.142305] LNetError: 22025:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (117): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4g2.ccs.ornl.gov kernel: [1915980.190848] LNetError: 22025:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4f4.ccs.ornl.gov kernel: [1915969.773706] LNetError: 22049:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4f4.ccs.ornl.gov kernel: [1915969.804490] LNetError: 22049:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4f4.ccs.ornl.gov kernel: [1915969.835057] LNetError: 22049:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4f4.ccs.ornl.gov kernel: [1915969.875217] LNetError: 22049:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4c3.ccs.ornl.gov kernel: [1915966.134978] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88099e8a8000 x1471904512246228/t0(0) o38->atlas2-MDT0000-lwp-OST032a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4b8.ccs.ornl.gov kernel: [1915971.212490] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8810114f3000 x1471904512564060/t0(0) o38->atlas2-MDT0000-lwp-OST03b7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4c3.ccs.ornl.gov kernel: [1915966.234011] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4b8.ccs.ornl.gov kernel: [1915971.311478] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4b7.ccs.ornl.gov kernel: [1915963.419589] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880a4e0bf400 x1471904509377440/t0(0) o38->atlas2-MDT0000-lwp-OST03b6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4c5.ccs.ornl.gov kernel: [1915965.103896] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8806abf38800 x1471904510171376/t0(0) o38->atlas2-MDT0000-lwp-OST032c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4f3.ccs.ornl.gov kernel: [1915964.045534] LNetError: 25894:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4c5.ccs.ornl.gov kernel: [1915965.202831] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4f3.ccs.ornl.gov kernel: [1915964.072669] LNetError: 25894:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4f3.ccs.ornl.gov kernel: [1915964.103493] LNetError: 25894:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4f3.ccs.ornl.gov kernel: [1915964.143531] LNetError: 25894:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3b2.ccs.ornl.gov kernel: [1915969.368704] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880d231d9800 x1471904511917152/t0(0) o38->atlas2-MDT0000-lwp-OST0369@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4b5.ccs.ornl.gov kernel: [1915986.927016] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8803eebd9000 x1471904518365808/t0(0) o38->atlas2-MDT0000-lwp-OST03b4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3b2.ccs.ornl.gov kernel: [1915969.458941] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4b5.ccs.ornl.gov kernel: [1915987.025593] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4e3.ccs.ornl.gov kernel: [1915975.177180] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8808487dc000 x1471904515523868/t0(0) o38->atlas2-MDT0000-lwp-OST03ca@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4g4.ccs.ornl.gov kernel: [1915983.771259] LNetError: 25705:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4g4.ccs.ornl.gov kernel: [1915983.801572] LNetError: 25705:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4g4.ccs.ornl.gov kernel: [1915983.832165] LNetError: 25705:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4g4.ccs.ornl.gov kernel: [1915983.872414] LNetError: 25705:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4e3.ccs.ornl.gov kernel: [1915975.276401] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3b4.ccs.ornl.gov kernel: [1915980.940871] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880f34f9b800 x1471904510548704/t0(0) o38->atlas2-MDT0000-lwp-OST036b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3b4.ccs.ornl.gov kernel: [1915981.039173] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3a2.ccs.ornl.gov kernel: [1915968.896539] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880fa2120400 x1471904510524428/t0(0) o38->atlas2-MDT0000-lwp-OST0361@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3a7.ccs.ornl.gov kernel: [1915967.494768] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88077e8a9c00 x1471904518655776/t0(0) o38->atlas2-MDT0000-lwp-OST0366@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3a2.ccs.ornl.gov kernel: [1915968.995564] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4f6.ccs.ornl.gov kernel: [1915964.093835] LNetError: 22175:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3a7.ccs.ornl.gov kernel: [1915967.594027] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4f6.ccs.ornl.gov kernel: [1915964.124625] LNetError: 22175:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4f6.ccs.ornl.gov kernel: [1915964.155309] LNetError: 22175:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4f6.ccs.ornl.gov kernel: [1915964.204078] LNetError: 22175:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4c1.ccs.ornl.gov kernel: [1915974.860585] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88017ff48800 x1471904516097656/t0(0) o38->atlas2-MDT0000-lwp-OST0328@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4d7.ccs.ornl.gov kernel: [1915970.062114] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880c7b15cc00 x1471904514720136/t0(0) o38->atlas2-MDT0000-lwp-OST03c6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4c1.ccs.ornl.gov kernel: [1915974.959569] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4d7.ccs.ornl.gov kernel: [1915970.160989] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4h6.ccs.ornl.gov kernel: [1915968.118229] LNetError: 25859:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 17 seconds Jul 17 17:06:54 atlas-oss4h6.ccs.ornl.gov kernel: [1915968.148516] LNetError: 25859:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4h6.ccs.ornl.gov kernel: [1915968.179439] LNetError: 25859:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (117): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4h6.ccs.ornl.gov kernel: [1915968.227959] LNetError: 25859:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4i4.ccs.ornl.gov kernel: [1915974.229008] LNetError: 22063:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4i4.ccs.ornl.gov kernel: [1915974.257477] LNetError: 22063:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4i4.ccs.ornl.gov kernel: [1915974.297483] LNetError: 22063:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (117): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4i4.ccs.ornl.gov kernel: [1915974.337883] LNetError: 22063:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4g8.ccs.ornl.gov kernel: [1915968.415177] LNetError: 25445:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4g8.ccs.ornl.gov kernel: [1915968.444445] LNetError: 25445:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4g8.ccs.ornl.gov kernel: [1915968.474998] LNetError: 25445:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4g8.ccs.ornl.gov kernel: [1915968.515286] LNetError: 25445:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3a4.ccs.ornl.gov kernel: [1915959.359698] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8801ecb2a000 x1471904512538932/t0(0) o38->atlas2-MDT0000-lwp-OST0363@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3c6.ccs.ornl.gov kernel: [1915961.627477] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880de8f56400 x1471904508187024/t0(0) o38->atlas2-MDT0000-lwp-OST0375@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3c6.ccs.ornl.gov kernel: [1915961.721311] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4d5.ccs.ornl.gov kernel: [1915981.878750] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880850b3a400 x1471904519136972/t0(0) o38->atlas2-MDT0000-lwp-OST03c4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4d5.ccs.ornl.gov kernel: [1915981.977750] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4h8.ccs.ornl.gov kernel: [1915960.259711] LNetError: 24516:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3e4.ccs.ornl.gov kernel: [1915983.686594] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4h8.ccs.ornl.gov kernel: [1915960.295382] LNetError: 24516:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4h8.ccs.ornl.gov kernel: [1915960.326112] LNetError: 24516:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4h8.ccs.ornl.gov kernel: [1915960.366475] LNetError: 24516:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4a1.ccs.ornl.gov kernel: [1915967.387617] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4g1.ccs.ornl.gov kernel: [1915966.410765] LNetError: 25702:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4g1.ccs.ornl.gov kernel: [1915966.440351] LNetError: 25702:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4g1.ccs.ornl.gov kernel: [1915966.471068] LNetError: 25702:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4g1.ccs.ornl.gov kernel: [1915966.511232] LNetError: 25702:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3c7.ccs.ornl.gov kernel: [1915970.254478] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88084f854400 x1471904510248824/t0(0) o38->atlas2-MDT0000-lwp-OST0376@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3g4.ccs.ornl.gov kernel: [1914579.839473] LNetError: 13502:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3g4.ccs.ornl.gov kernel: [1914579.863435] LNetError: 13502:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3g4.ccs.ornl.gov kernel: [1914579.893984] LNetError: 13502:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss3g4.ccs.ornl.gov kernel: [1914579.934145] LNetError: 13502:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4h4.ccs.ornl.gov kernel: [1915988.754287] LNetError: 25723:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4h4.ccs.ornl.gov kernel: [1915988.777461] LNetError: 25723:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4h4.ccs.ornl.gov kernel: [1915988.808028] LNetError: 25723:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4h4.ccs.ornl.gov kernel: [1915988.848081] LNetError: 25723:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3c8.ccs.ornl.gov kernel: [1915974.430082] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88103a168c00 x1471904507928192/t0(0) o38->atlas2-MDT0000-lwp-OST0377@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4h3.ccs.ornl.gov kernel: [1915982.025607] LNetError: 25950:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3d3.ccs.ornl.gov kernel: [1915976.210657] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8806e7fb9400 x1471904510125868/t0(0) o38->atlas2-MDT0000-lwp-OST037a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3d3.ccs.ornl.gov kernel: [1915976.300931] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4h3.ccs.ornl.gov kernel: [1915982.054079] LNetError: 25950:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4h3.ccs.ornl.gov kernel: [1915982.084798] LNetError: 25950:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4h3.ccs.ornl.gov kernel: [1915982.124943] LNetError: 25950:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3b1.ccs.ornl.gov kernel: [1915976.101463] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4e2.ccs.ornl.gov kernel: [1915983.181285] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880b5db68400 x1471904514146492/t0(0) o38->atlas2-MDT0000-lwp-OST03c9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4e2.ccs.ornl.gov kernel: [1915983.280245] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4f2.ccs.ornl.gov kernel: [1915972.898172] LNetError: 25615:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4f2.ccs.ornl.gov kernel: [1915972.929674] LNetError: 25615:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4f2.ccs.ornl.gov kernel: [1915972.960601] LNetError: 25615:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4f2.ccs.ornl.gov kernel: [1915973.000747] LNetError: 25615:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4f7.ccs.ornl.gov kernel: [1915968.407086] LNetError: 25868:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 17 seconds Jul 17 17:06:54 atlas-oss4f7.ccs.ornl.gov kernel: [1915968.431830] LNetError: 25868:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4f7.ccs.ornl.gov kernel: [1915968.462157] LNetError: 25868:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (117): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4f7.ccs.ornl.gov kernel: [1915968.510925] LNetError: 25868:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3a3.ccs.ornl.gov kernel: [1915966.615728] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880d1cb5f000 x1471904526380064/t0(0) o38->atlas2-MDT0000-lwp-OST0362@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3a3.ccs.ornl.gov kernel: [1915966.714840] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4h7.ccs.ornl.gov kernel: [1915977.621364] LNetError: 25998:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 17 seconds Jul 17 17:06:54 atlas-oss4h7.ccs.ornl.gov kernel: [1915977.648656] LNetError: 25998:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4h7.ccs.ornl.gov kernel: [1915977.679311] LNetError: 25998:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (117): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4h7.ccs.ornl.gov kernel: [1915977.727909] LNetError: 25998:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3a6.ccs.ornl.gov kernel: [1915972.006888] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8803d287f400 x1471904512394792/t0(0) o38->atlas2-MDT0000-lwp-OST0365@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3a6.ccs.ornl.gov kernel: [1915972.105935] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3e6.ccs.ornl.gov kernel: [1915966.960381] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880df8588400 x1471904520534784/t0(0) o38->atlas2-MDT0000-lwp-OST0385@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3e6.ccs.ornl.gov kernel: [1915967.050271] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4g6.ccs.ornl.gov kernel: [1915979.058126] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8804d5a70c00 x1471904507898840/t0(0) o38->atlas2-MDT0000-lwp-OST03dd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4g6.ccs.ornl.gov kernel: [1915979.157224] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4i7.ccs.ornl.gov kernel: [1915982.661796] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631096/real 1405631214] req@ffff880ab3f2d400 x1471904511375264/t0(0) o38->atlas2-MDT0000-lwp-OST035e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631376 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4i7.ccs.ornl.gov kernel: [1915982.760824] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4d8.ccs.ornl.gov kernel: [1915982.438673] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880292ccc400 x1471904516223140/t0(0) o38->atlas2-MDT0000-lwp-OST03c7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4i5.ccs.ornl.gov kernel: [1915977.296561] LNetError: 25853:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4d8.ccs.ornl.gov kernel: [1915982.537665] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4i5.ccs.ornl.gov kernel: [1915977.325960] LNetError: 25853:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4i5.ccs.ornl.gov kernel: [1915977.356488] LNetError: 25853:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4i5.ccs.ornl.gov kernel: [1915977.396802] LNetError: 25853:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3d2.ccs.ornl.gov kernel: [1915971.097681] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff881006057c00 x1471904511214796/t0(0) o38->atlas2-MDT0000-lwp-OST0379@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3d2.ccs.ornl.gov kernel: [1915971.196287] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4d6.ccs.ornl.gov kernel: [1915969.541043] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4e1.ccs.ornl.gov kernel: [1915967.245605] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880f9a425400 x1471904515144904/t0(0) o38->atlas2-MDT0000-lwp-OST03c8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4g3.ccs.ornl.gov kernel: [1915980.487086] LNetError: 25713:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4e1.ccs.ornl.gov kernel: [1915967.344575] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3c5.ccs.ornl.gov kernel: [1915976.066607] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4g3.ccs.ornl.gov kernel: [1915980.516028] LNetError: 25713:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4g3.ccs.ornl.gov kernel: [1915980.546633] LNetError: 25713:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4g3.ccs.ornl.gov kernel: [1915980.586970] LNetError: 25713:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3a1.ccs.ornl.gov kernel: [1915964.217225] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880ad3022400 x1471904525931912/t0(0) o38->atlas2-MDT0000-lwp-OST0360@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3a1.ccs.ornl.gov kernel: [1915964.307270] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3h6.ccs.ornl.gov kernel: [1915969.773993] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880dc2b25800 x1471904512216672/t0(0) o38->atlas2-MDT0000-lwp-OST039d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3g1.ccs.ornl.gov kernel: [1914559.350448] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880834963400 x1471904510299780/t0(0) o38->atlas2-MDT0000-lwp-OST0390@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3g1.ccs.ornl.gov kernel: [1914559.449774] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3a5.ccs.ornl.gov kernel: [1915962.916875] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88073adfa400 x1471904511738612/t0(0) o38->atlas2-MDT0000-lwp-OST02d4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3a5.ccs.ornl.gov kernel: [1915963.016069] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3h2.ccs.ornl.gov kernel: [1915972.439922] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8808d15f1000 x1471904518760656/t0(0) o38->atlas2-MDT0000-lwp-OST0399@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4b7.ccs.ornl.gov kernel: [1915963.518742] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3h2.ccs.ornl.gov kernel: [1915972.530555] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3g2.ccs.ornl.gov kernel: [1914566.025427] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8801c0d26c00 x1471904509036272/t0(0) o38->atlas2-MDT0000-lwp-OST0391@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3g2.ccs.ornl.gov kernel: [1914566.124713] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3f1.ccs.ornl.gov kernel: [1915973.239549] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8802be6e0000 x1471904508988576/t0(0) o38->atlas2-MDT0000-lwp-OST0388@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3f1.ccs.ornl.gov kernel: [1915973.330333] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4h5.ccs.ornl.gov kernel: [1915971.907637] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8802bdc7ac00 x1471904509313612/t0(0) o38->atlas2-MDT0000-lwp-OST03e4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3g5.ccs.ornl.gov kernel: [1914570.978628] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88067ef5f000 x1471904518249236/t0(0) o38->atlas2-MDT0000-lwp-OST0394@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4h5.ccs.ornl.gov kernel: [1915971.998658] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3g5.ccs.ornl.gov kernel: [1914571.077849] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3e5.ccs.ornl.gov kernel: [1915970.650150] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8809f6694c00 x1471904510654952/t0(0) o38->atlas2-MDT0000-lwp-OST0384@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3e5.ccs.ornl.gov kernel: [1915970.740789] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4h1.ccs.ornl.gov kernel: [1915963.239530] LNetError: 25754:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss4h1.ccs.ornl.gov kernel: [1915963.265653] LNetError: 25754:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss4h1.ccs.ornl.gov kernel: [1915963.296385] LNetError: 25754:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4h1.ccs.ornl.gov kernel: [1915963.344978] LNetError: 25754:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 2 previous similar messages Jul 17 17:06:54 atlas-oss3g6.ccs.ornl.gov kernel: [1914577.785225] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880bb6d4b000 x1471904523473976/t0(0) o38->atlas2-MDT0000-lwp-OST0395@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3i3.ccs.ornl.gov kernel: [1915964.874798] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880cbca8f800 x1471904518169420/t0(0) o38->atlas2-MDT0000-lwp-OST03a2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4i8.ccs.ornl.gov kernel: [1915983.053393] LNetError: 25986:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds Jul 17 17:06:54 atlas-oss3i3.ccs.ornl.gov kernel: [1915964.973692] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4i8.ccs.ornl.gov kernel: [1915983.095254] LNetError: 25986:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4i8.ccs.ornl.gov kernel: [1915983.126006] LNetError: 25986:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (116): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4i8.ccs.ornl.gov kernel: [1915983.166089] LNetError: 25986:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss3b3.ccs.ornl.gov kernel: [1915962.381591] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8806fd2d9c00 x1471904513820576/t0(0) o38->atlas2-MDT0000-lwp-OST036a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3b3.ccs.ornl.gov kernel: [1915962.467842] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3g3.ccs.ornl.gov kernel: [1914566.800477] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8804890dd800 x1471904516976908/t0(0) o38->atlas2-MDT0000-lwp-OST0392@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3i2.ccs.ornl.gov kernel: [1915975.334581] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8801a59f3c00 x1471904514016408/t0(0) o38->atlas2-MDT0000-lwp-OST03a1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3g3.ccs.ornl.gov kernel: [1914566.891075] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3i2.ccs.ornl.gov kernel: [1915975.425220] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3h3.ccs.ornl.gov kernel: [1915964.994497] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8803c10b6800 x1471904512413756/t0(0) o38->atlas2-MDT0000-lwp-OST039a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3h3.ccs.ornl.gov kernel: [1915965.094081] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4i2.ccs.ornl.gov kernel: [1915992.233560] LNetError: 25638:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 17 seconds Jul 17 17:06:54 atlas-oss3a4.ccs.ornl.gov kernel: [1915959.450713] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4i2.ccs.ornl.gov kernel: [1915992.260263] LNetError: 25638:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4i2.ccs.ornl.gov kernel: [1915992.290936] LNetError: 25638:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (117): c: 0, oc: 0, rc: 63 Jul 17 17:06:54 atlas-oss4i2.ccs.ornl.gov kernel: [1915992.339719] LNetError: 25638:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Skipped 1 previous similar message Jul 17 17:06:54 atlas-oss4i3.ccs.ornl.gov kernel: [1915961.636132] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880932f0cc00 x1471904523159736/t0(0) o38->atlas2-MDT0000-lwp-OST03ea@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4i3.ccs.ornl.gov kernel: [1915961.734862] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4h2.ccs.ornl.gov kernel: [1915990.632312] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631096/real 1405631214] req@ffff8803a38b5c00 x1471904508411292/t0(0) o38->atlas2-MDT0000-lwp-OST03e1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631376 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4g7.ccs.ornl.gov kernel: [1915991.706636] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88100183dc00 x1471904508873728/t0(0) o38->atlas2-MDT0000-lwp-OST03de@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4h2.ccs.ornl.gov kernel: [1915990.731354] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4g7.ccs.ornl.gov kernel: [1915991.805892] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3h5.ccs.ornl.gov kernel: [1915989.116644] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88054a492c00 x1471904507806740/t0(0) o38->atlas2-MDT0000-lwp-OST039c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3h5.ccs.ornl.gov kernel: [1915989.215850] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3h8.ccs.ornl.gov kernel: [1915966.274521] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880675c30400 x1471904514772972/t0(0) o38->atlas2-MDT0000-lwp-OST039f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3h8.ccs.ornl.gov kernel: [1915966.373338] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3c7.ccs.ornl.gov kernel: [1915970.345278] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4g5.ccs.ornl.gov kernel: [1915969.515700] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631096/real 1405631214] req@ffff8805a1713000 x1471904510456896/t0(0) o38->atlas2-MDT0000-lwp-OST034c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631376 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4g5.ccs.ornl.gov kernel: [1915969.614838] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3c8.ccs.ornl.gov kernel: [1915974.520755] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3i7.ccs.ornl.gov kernel: [1915966.558718] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff881008feac00 x1471904508389448/t0(0) o38->atlas2-MDT0000-lwp-OST03a6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3i7.ccs.ornl.gov kernel: [1915966.657595] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4b1.ccs.ornl.gov kernel: [1915973.904798] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8807d77e9400 x1471904507949240/t0(0) o38->atlas2-MDT0000-lwp-OST03b0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4b1.ccs.ornl.gov kernel: [1915974.003910] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4b4.ccs.ornl.gov kernel: [1915967.533608] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88066147d000 x1471904528872164/t0(0) o38->atlas2-MDT0000-lwp-OST03b3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3h4.ccs.ornl.gov kernel: [1915973.192426] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8810714dfc00 x1471904509245412/t0(0) o38->atlas2-MDT0000-lwp-OST039b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4a7.ccs.ornl.gov kernel: [1915981.076481] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8802ff83ec00 x1471904510936168/t0(0) o38->atlas2-MDT0000-lwp-OST03ae@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4i6.ccs.ornl.gov kernel: [1915982.798846] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88082ce87c00 x1471904510597728/t0(0) o38->atlas2-MDT0000-lwp-OST035d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4i6.ccs.ornl.gov kernel: [1915982.897626] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3h1.ccs.ornl.gov kernel: [1915958.130911] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880574476800 x1471904517766048/t0(0) o38->atlas2-MDT0000-lwp-OST0398@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3i6.ccs.ornl.gov kernel: [1915968.276233] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880591031800 x1471904509634152/t0(0) o38->atlas2-MDT0000-lwp-OST03a5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3i6.ccs.ornl.gov kernel: [1915968.375761] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3g8.ccs.ornl.gov kernel: [1914567.463708] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88050afd0400 x1471904513602568/t0(0) o38->atlas2-MDT0000-lwp-OST0397@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4i1.ccs.ornl.gov kernel: [1915969.537616] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8802bbef6000 x1471904513319876/t0(0) o38->atlas2-MDT0000-lwp-OST0358@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3g8.ccs.ornl.gov kernel: [1914567.562855] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4i1.ccs.ornl.gov kernel: [1915969.636580] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4a5.ccs.ornl.gov kernel: [1915988.389996] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8809b64e9800 x1471904519454296/t0(0) o38->atlas2-MDT0000-lwp-OST03ac@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4a3.ccs.ornl.gov kernel: [1915985.384589] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88016209e000 x1471904514689040/t0(0) o38->atlas2-MDT0000-lwp-OST03aa@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4a5.ccs.ornl.gov kernel: [1915988.480560] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4a3.ccs.ornl.gov kernel: [1915985.483477] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4a2.ccs.ornl.gov kernel: [1915984.050225] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8805dfb87c00 x1471904510895860/t0(0) o38->atlas2-MDT0000-lwp-OST03a9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4a2.ccs.ornl.gov kernel: [1915984.149053] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3e8.ccs.ornl.gov kernel: [1915969.542059] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880310b81c00 x1471904515139732/t0(0) o38->atlas2-MDT0000-lwp-OST0387@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3e8.ccs.ornl.gov kernel: [1915969.641291] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4c6.ccs.ornl.gov kernel: [1915970.344645] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88094c9d5800 x1471904517937372/t0(0) o38->atlas2-MDT0000-lwp-OST03bd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4b3.ccs.ornl.gov kernel: [1915963.721514] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631096/real 1405631214] req@ffff8802429bb800 x1471904509180676/t0(0) o38->atlas2-MDT0000-lwp-OST03b2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631376 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4c6.ccs.ornl.gov kernel: [1915970.434976] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4a4.ccs.ornl.gov kernel: [1915970.973055] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8802e94d5c00 x1471904509591248/t0(0) o38->atlas2-MDT0000-lwp-OST03ab@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4a4.ccs.ornl.gov kernel: [1915971.072193] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3i8.ccs.ornl.gov kernel: [1915983.321775] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8808309e0400 x1471904515355620/t0(0) o38->atlas2-MDT0000-lwp-OST03a7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3i1.ccs.ornl.gov kernel: [1915969.573009] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88054204c000 x1471904519755548/t0(0) o38->atlas2-MDT0000-lwp-OST0310@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3i8.ccs.ornl.gov kernel: [1915983.420572] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3i1.ccs.ornl.gov kernel: [1915969.671896] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3i4.ccs.ornl.gov kernel: [1915975.293280] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880465cb3000 x1471904510470528/t0(0) o38->atlas2-MDT0000-lwp-OST03a3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4d1.ccs.ornl.gov kernel: [1915965.464566] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff881009ab8000 x1471904516811856/t0(0) o38->atlas2-MDT0000-lwp-OST03c0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4d1.ccs.ornl.gov kernel: [1915965.555199] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3h6.ccs.ornl.gov kernel: [1915969.864762] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4e7.ccs.ornl.gov kernel: [1915966.832014] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8806e8ab8000 x1471904522621664/t0(0) o38->atlas2-MDT0000-lwp-OST03ce@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4e7.ccs.ornl.gov kernel: [1915966.932231] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4a8.ccs.ornl.gov kernel: [1915975.479028] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88100f653800 x1471904507750912/t0(0) o38->atlas2-MDT0000-lwp-OST03af@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4a8.ccs.ornl.gov kernel: [1915975.577986] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4e6.ccs.ornl.gov kernel: [1915985.196852] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880ff9a41400 x1471904508408268/t0(0) o38->atlas2-MDT0000-lwp-OST03cd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4e6.ccs.ornl.gov kernel: [1915985.287283] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4c8.ccs.ornl.gov kernel: [1915961.290513] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880c8b56e400 x1471904512197068/t0(0) o38->atlas2-MDT0000-lwp-OST03bf@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4d3.ccs.ornl.gov kernel: [1915965.898383] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880fccb8f400 x1471904516978416/t0(0) o38->atlas2-MDT0000-lwp-OST03c2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4d3.ccs.ornl.gov kernel: [1915965.997598] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4d4.ccs.ornl.gov kernel: [1915989.771933] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8806e8124c00 x1471904513679304/t0(0) o38->atlas2-MDT0000-lwp-OST03c3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4d4.ccs.ornl.gov kernel: [1915989.870794] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4f1.ccs.ornl.gov kernel: [1915973.617326] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88062c3c3000 x1471904513371212/t0(0) o38->atlas2-MDT0000-lwp-OST03d0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4f1.ccs.ornl.gov kernel: [1915973.708062] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3g6.ccs.ornl.gov kernel: [1914577.876193] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4a6.ccs.ornl.gov kernel: [1915971.507234] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88029b5f6400 x1471904512502920/t0(0) o38->atlas2-MDT0000-lwp-OST03ad@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4a6.ccs.ornl.gov kernel: [1915971.607326] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4d2.ccs.ornl.gov kernel: [1915974.538111] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631096/real 1405631214] req@ffff88036438d400 x1471904512221116/t0(0) o38->atlas2-MDT0000-lwp-OST03c1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631376 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4d2.ccs.ornl.gov kernel: [1915974.637015] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4f5.ccs.ornl.gov kernel: [1915967.378842] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880df9ed1c00 x1471904509391080/t0(0) o38->atlas2-MDT0000-lwp-OST03d4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4f5.ccs.ornl.gov kernel: [1915967.464372] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3h4.ccs.ornl.gov kernel: [1915973.291711] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4a7.ccs.ornl.gov kernel: [1915981.175826] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4f4.ccs.ornl.gov kernel: [1915969.914622] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8804cfb5d000 x1471904516980408/t0(0) o38->atlas2-MDT0000-lwp-OST0343@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4f4.ccs.ornl.gov kernel: [1915970.005409] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4f8.ccs.ornl.gov kernel: [1915968.704365] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8802a2272400 x1471904511806176/t0(0) o38->atlas2-MDT0000-lwp-OST03d7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4b3.ccs.ornl.gov kernel: [1915963.820789] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4b4.ccs.ornl.gov kernel: [1915967.624408] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4e8.ccs.ornl.gov kernel: [1915982.412473] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8806df5f8400 x1471904518472944/t0(0) o38->atlas2-MDT0000-lwp-OST03cf@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3h1.ccs.ornl.gov kernel: [1915958.221576] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4e8.ccs.ornl.gov kernel: [1915982.511561] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4g2.ccs.ornl.gov kernel: [1915980.221751] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631096/real 1405631214] req@ffff880ba3170400 x1471904509229128/t0(0) o38->atlas2-MDT0000-lwp-OST03d9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631376 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3i4.ccs.ornl.gov kernel: [1915975.392092] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4g8.ccs.ornl.gov kernel: [1915968.554506] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8803286fd400 x1471904521746976/t0(0) o38->atlas2-MDT0000-lwp-OST03df@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4f3.ccs.ornl.gov kernel: [1915964.182973] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88020306ac00 x1471904513977296/t0(0) o38->atlas2-MDT0000-lwp-OST03d2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4f3.ccs.ornl.gov kernel: [1915964.282176] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4h8.ccs.ornl.gov kernel: [1915960.405702] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880539f92400 x1471904514267352/t0(0) o38->atlas2-MDT0000-lwp-OST03e7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4h8.ccs.ornl.gov kernel: [1915960.496310] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4g1.ccs.ornl.gov kernel: [1915966.550769] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8801a9532400 x1471904508970544/t0(0) o38->atlas2-MDT0000-lwp-OST03d8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4g1.ccs.ornl.gov kernel: [1915966.641250] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4c8.ccs.ornl.gov kernel: [1915961.389606] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss3g4.ccs.ornl.gov kernel: [1914579.973545] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880e9f0d5400 x1471904510791508/t0(0) o38->atlas2-MDT0000-lwp-OST0393@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss3g4.ccs.ornl.gov kernel: [1914580.063902] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4g4.ccs.ornl.gov kernel: [1915983.911714] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880104d43c00 x1471904515530820/t0(0) o38->atlas2-MDT0000-lwp-OST03db@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4f6.ccs.ornl.gov kernel: [1915964.235229] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880a5962e000 x1471904511708708/t0(0) o38->atlas2-MDT0000-lwp-OST0345@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4f6.ccs.ornl.gov kernel: [1915964.334505] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4h3.ccs.ornl.gov kernel: [1915982.164610] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880d684d9000 x1471904509650992/t0(0) o38->atlas2-MDT0000-lwp-OST03e2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4h6.ccs.ornl.gov kernel: [1915968.259341] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631096/real 1405631214] req@ffff880b18c1b000 x1471904515532256/t0(0) o38->atlas2-MDT0000-lwp-OST03e5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631376 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4h6.ccs.ornl.gov kernel: [1915968.358221] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4i4.ccs.ornl.gov kernel: [1915974.368769] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880c7feff000 x1471904516513584/t0(0) o38->atlas2-MDT0000-lwp-OST03eb@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4i4.ccs.ornl.gov kernel: [1915974.467847] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4f2.ccs.ornl.gov kernel: [1915973.040096] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8808bc1b8400 x1471904508910836/t0(0) o38->atlas2-MDT0000-lwp-OST03d1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4f2.ccs.ornl.gov kernel: [1915973.130825] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4h4.ccs.ornl.gov kernel: [1915988.878888] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8805c456c800 x1471904512334512/t0(0) o38->atlas2-MDT0000-lwp-OST03e3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4h4.ccs.ornl.gov kernel: [1915988.978272] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4f7.ccs.ornl.gov kernel: [1915968.542230] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631096/real 1405631214] req@ffff8802e6ec9000 x1471904510455200/t0(0) o38->atlas2-MDT0000-lwp-OST0346@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631376 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4f7.ccs.ornl.gov kernel: [1915968.641185] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4h7.ccs.ornl.gov kernel: [1915977.759089] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631096/real 1405631214] req@ffff8806c4c9c000 x1471904512615304/t0(0) o38->atlas2-MDT0000-lwp-OST0356@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631376 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4h7.ccs.ornl.gov kernel: [1915977.858007] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4f8.ccs.ornl.gov kernel: [1915968.804008] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4i5.ccs.ornl.gov kernel: [1915977.436083] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff88051089d800 x1471904512720556/t0(0) o38->atlas2-MDT0000-lwp-OST03ec@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4i5.ccs.ornl.gov kernel: [1915977.535275] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4g3.ccs.ornl.gov kernel: [1915980.626172] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8808e19b2000 x1471904517610380/t0(0) o38->atlas2-MDT0000-lwp-OST03da@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4g3.ccs.ornl.gov kernel: [1915980.725355] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4g2.ccs.ornl.gov kernel: [1915980.320968] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4g4.ccs.ornl.gov kernel: [1915984.011138] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4h1.ccs.ornl.gov kernel: [1915963.376029] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff880dea9ef400 x1471904511603272/t0(0) o38->atlas2-MDT0000-lwp-OST0350@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4h1.ccs.ornl.gov kernel: [1915963.475048] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4g8.ccs.ornl.gov kernel: [1915968.645419] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4i8.ccs.ornl.gov kernel: [1915983.197368] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631214] req@ffff8805f0f05c00 x1471904520247276/t0(0) o38->atlas2-MDT0000-lwp-OST03ef@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4i8.ccs.ornl.gov kernel: [1915983.296082] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4h3.ccs.ornl.gov kernel: [1915982.255123] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:54 atlas-oss4i2.ccs.ornl.gov kernel: [1915992.370622] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631096/real 1405631214] req@ffff8802f220d800 x1471904514634688/t0(0) o38->atlas2-MDT0000-lwp-OST03e9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631376 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:54 atlas-oss4i2.ccs.ornl.gov kernel: [1915992.470709] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:56 atlas-oss3f5.ccs.ornl.gov kernel: [802663.668378] LNetError: 79083:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 19 seconds Jul 17 17:06:56 atlas-oss3f5.ccs.ornl.gov kernel: [802663.693566] LNetError: 79083:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (119): c: 0, oc: 0, rc: 63 Jul 17 17:06:56 atlas-oss3f6.ccs.ornl.gov kernel: [802676.356645] LNetError: 79550:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 19 seconds Jul 17 17:06:56 atlas-oss3f6.ccs.ornl.gov kernel: [802676.384995] LNetError: 79550:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (119): c: 0, oc: 0, rc: 63 Jul 17 17:06:56 atlas-oss3f8.ccs.ornl.gov kernel: [802674.170225] LNetError: 80971:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 19 seconds Jul 17 17:06:56 atlas-oss3f8.ccs.ornl.gov kernel: [802674.201749] LNetError: 80971:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (119): c: 0, oc: 0, rc: 63 Jul 17 17:06:56 atlas-oss3f5.ccs.ornl.gov kernel: [802663.734707] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631216] req@ffff880ff1489400 x1473076856406440/t0(0) o38->atlas2-MDT0000-lwp-OST02fc@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:56 atlas-oss3f5.ccs.ornl.gov kernel: [802663.833661] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:56 atlas-oss3f6.ccs.ornl.gov kernel: [802676.425818] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631216] req@ffff880bbbb51000 x1473076856558736/t0(0) o38->atlas2-MDT0000-lwp-OST038d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:56 atlas-oss3f8.ccs.ornl.gov kernel: [802674.242220] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631216] req@ffff880fc15ccc00 x1473076861321092/t0(0) o38->atlas2-MDT0000-lwp-OST038f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:56 atlas-oss3f8.ccs.ornl.gov kernel: [802674.341639] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:56 atlas-oss3f6.ccs.ornl.gov kernel: [802676.524812] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:06:56 atlas-oss3f7.ccs.ornl.gov kernel: [802303.000464] LNetError: 78551:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 19 seconds Jul 17 17:06:56 atlas-oss3f7.ccs.ornl.gov kernel: [802303.025377] LNetError: 78551:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (119): c: 0, oc: 0, rc: 63 Jul 17 17:06:56 atlas-oss3f7.ccs.ornl.gov kernel: [802303.065985] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631097/real 1405631216] req@ffff8801a984c400 x1473076863306340/t0(0) o38->atlas2-MDT0000-lwp-OST038e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631377 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:06:56 atlas-oss3f7.ccs.ornl.gov kernel: [802303.165247] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:07:48 atlas-oss3i5.ccs.ornl.gov kernel: [702331.427585] LNetError: 17755:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 24 seconds Jul 17 17:07:48 atlas-oss3a8.ccs.ornl.gov kernel: [702332.308792] LNetError: 17719:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 24 seconds Jul 17 17:07:48 atlas-oss3a8.ccs.ornl.gov kernel: [702332.333082] LNetError: 17719:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (124): c: 0, oc: 0, rc: 63 Jul 17 17:07:48 atlas-oss3i5.ccs.ornl.gov kernel: [702331.450693] LNetError: 17755:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (124): c: 0, oc: 0, rc: 63 Jul 17 17:07:48 atlas-oss3d4.ccs.ornl.gov kernel: [702331.751188] LNetError: 16375:0:(o2iblnd_cb.c:3012:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 24 seconds Jul 17 17:07:48 atlas-oss3d4.ccs.ornl.gov kernel: [702331.781646] LNetError: 16375:0:(o2iblnd_cb.c:3075:kiblnd_check_conns()) Timed out RDMA with 10.36.226.74@o2ib (124): c: 0, oc: 0, rc: 63 Jul 17 17:07:48 atlas-oss3i5.ccs.ornl.gov kernel: [702331.491668] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631143/real 1405631268] req@ffff8801a426ac00 x1473175721933480/t0(0) o38->atlas2-MDT0000-lwp-OST0314@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631423 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:07:48 atlas-oss3a8.ccs.ornl.gov kernel: [702332.373943] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631143/real 1405631268] req@ffff88086f3ad000 x1473175720195064/t0(0) o38->atlas2-MDT0000-lwp-OST0367@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631423 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:07:48 atlas-oss3d4.ccs.ornl.gov kernel: [702331.822413] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631143/real 1405631268] req@ffff88083e7bac00 x1473175723812860/t0(0) o38->atlas2-MDT0000-lwp-OST037b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631423 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:07:48 atlas-oss3d4.ccs.ornl.gov kernel: [702331.921359] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:07:48 atlas-oss3i5.ccs.ornl.gov kernel: [702331.590711] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:07:48 atlas-oss3a8.ccs.ornl.gov kernel: [702332.473307] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:09:07 atlas-oss4g5.ccs.ornl.gov kernel: [1916102.467883] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631246/real 1405631346] req@ffff88021ee56800 x1471904510456904/t0(0) o38->atlas2-MDT0000-lwp-OST034c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631526 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4g2.ccs.ornl.gov kernel: [1916113.143162] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631246/real 1405631346] req@ffff880e0b73e000 x1471904509229136/t0(0) o38->atlas2-MDT0000-lwp-OST03d9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631526 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4h7.ccs.ornl.gov kernel: [1916110.662917] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631246/real 1405631346] req@ffff8807d0d27000 x1471904512615312/t0(0) o38->atlas2-MDT0000-lwp-OST03e6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631526 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4d2.ccs.ornl.gov kernel: [1916107.481047] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631246/real 1405631347] req@ffff88072d45b400 x1471904512221124/t0(0) o38->atlas2-MDT0000-lwp-OST03c1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631526 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4i7.ccs.ornl.gov kernel: [1916115.673034] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631246/real 1405631347] req@ffff8809c5a34c00 x1471904511375272/t0(0) o38->atlas2-MDT0000-lwp-OST035e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631526 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4i2.ccs.ornl.gov kernel: [1916125.263350] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631246/real 1405631347] req@ffff880688df3400 x1471904514634696/t0(0) o38->atlas2-MDT0000-lwp-OST03e9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631526 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4h6.ccs.ornl.gov kernel: [1916101.186140] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631246/real 1405631347] req@ffff880def5cd000 x1471904515532264/t0(0) o38->atlas2-MDT0000-lwp-OST03e5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631526 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4b3.ccs.ornl.gov kernel: [1916096.698665] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631246/real 1405631347] req@ffff8803ce8d5400 x1471904509180684/t0(0) o38->atlas2-MDT0000-lwp-OST03b2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631526 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4h2.ccs.ornl.gov kernel: [1916123.637678] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631246/real 1405631347] req@ffff880552b3e000 x1471904508411300/t0(0) o38->atlas2-MDT0000-lwp-OST03e1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631526 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4f7.ccs.ornl.gov kernel: [1916101.470261] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631246/real 1405631347] req@ffff8804f647f400 x1471904510455208/t0(0) o38->atlas2-MDT0000-lwp-OST02b6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631526 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4f7.ccs.ornl.gov kernel: [1916101.562036] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 17:09:07 atlas-oss4h7.ccs.ornl.gov kernel: [1916110.749156] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss4b2.ccs.ornl.gov kernel: [1916125.235722] Lustre: 27461:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880b06721400 x1471904518026008/t0(0) o38->atlas2-MDT0000-lwp-OST03b1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4b2.ccs.ornl.gov kernel: [1916125.320529] Lustre: 27461:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 17 17:09:07 atlas-oss3e3.ccs.ornl.gov kernel: [1916099.121968] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880871beb800 x1471904510468612/t0(0) o38->atlas2-MDT0000-lwp-OST0382@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4i3.ccs.ornl.gov kernel: [1916094.685886] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880a79483000 x1471904523159744/t0(0) o38->atlas2-MDT0000-lwp-OST035a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4d7.ccs.ornl.gov kernel: [1916103.160577] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880f5c3b7c00 x1471904514720144/t0(0) o38->atlas2-MDT0000-lwp-OST03c6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3h5.ccs.ornl.gov kernel: [1916122.176542] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88082625b000 x1471904507806748/t0(0) o38->atlas2-MDT0000-lwp-OST039c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3f6.ccs.ornl.gov kernel: [802807.055343] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880e42db3800 x1473076856558744/t0(0) o38->atlas2-MDT0000-lwp-OST038d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4i4.ccs.ornl.gov kernel: [1916107.374721] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880982fa5800 x1471904516513592/t0(0) o38->atlas2-MDT0000-lwp-OST03eb@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3c1.ccs.ornl.gov kernel: [1916104.608169] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88018ab65000 x1471904511393132/t0(0) o38->atlas2-MDT0000-lwp-OST02e0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3i7.ccs.ornl.gov kernel: [1916099.627592] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8810031d0c00 x1471904508389456/t0(0) o38->atlas2-MDT0000-lwp-OST0316@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3i7.ccs.ornl.gov kernel: [1916099.717703] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss4i3.ccs.ornl.gov kernel: [1916094.775044] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss4f2.ccs.ornl.gov kernel: [1916106.050746] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880fb24cec00 x1471904508910844/t0(0) o38->atlas2-MDT0000-lwp-OST03d1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4d3.ccs.ornl.gov kernel: [1916098.957763] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880d982a6c00 x1471904516978424/t0(0) o38->atlas2-MDT0000-lwp-OST02a2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4h4.ccs.ornl.gov kernel: [1916121.907699] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8804d1d14800 x1471904512334520/t0(0) o38->atlas2-MDT0000-lwp-OST03e3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4c5.ccs.ornl.gov kernel: [1916098.241276] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880551dc3c00 x1471904510171384/t0(0) o38->atlas2-MDT0000-lwp-OST032c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4c3.ccs.ornl.gov kernel: [1916099.280944] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880b88e8ac00 x1471904512246236/t0(0) o38->atlas2-MDT0000-lwp-OST032a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3g8.ccs.ornl.gov kernel: [1914700.557919] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8807036bc000 x1471904513602576/t0(0) o38->atlas2-MDT0000-lwp-OST0397@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3h8.ccs.ornl.gov kernel: [1916099.376733] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8804f0cda800 x1471904514772980/t0(0) o38->atlas2-MDT0000-lwp-OST039f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4a4.ccs.ornl.gov kernel: [1916104.065657] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88075087c800 x1471904509591256/t0(0) o38->atlas2-MDT0000-lwp-OST03ab@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3d6.ccs.ornl.gov kernel: [1916116.902315] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8808cb007400 x1471904511183540/t0(0) o38->atlas2-MDT0000-lwp-OST037d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4d3.ccs.ornl.gov kernel: [1916099.048559] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss3e1.ccs.ornl.gov kernel: [1916105.280489] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8806370f7400 x1471904509694504/t0(0) o38->atlas2-MDT0000-lwp-OST02f0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4a5.ccs.ornl.gov kernel: [1916121.489100] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880d0a09ec00 x1471904519454304/t0(0) o38->atlas2-MDT0000-lwp-OST03ac@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4e7.ccs.ornl.gov kernel: [1916099.943954] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880871b96c00 x1471904522621672/t0(0) o38->atlas2-MDT0000-lwp-OST033e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4e7.ccs.ornl.gov kernel: [1916100.031477] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss3f5.ccs.ornl.gov kernel: [802794.434167] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880d69513c00 x1473076856406452/t0(0) o38->atlas2-MDT0000-lwp-OST02fc@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3f5.ccs.ornl.gov kernel: [802794.522580] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss4c8.ccs.ornl.gov kernel: [1916094.395172] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880af50f5800 x1471904512197076/t0(0) o38->atlas2-MDT0000-lwp-OST03bf@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4a2.ccs.ornl.gov kernel: [1916117.168523] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880283167000 x1471904510895868/t0(0) o38->atlas2-MDT0000-lwp-OST03a9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3f8.ccs.ornl.gov kernel: [802804.939484] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880eca6e0400 x1473076861321100/t0(0) o38->atlas2-MDT0000-lwp-OST038f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4c2.ccs.ornl.gov kernel: [1916106.485740] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88091454e800 x1471904508570824/t0(0) o38->atlas2-MDT0000-lwp-OST03b9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3g5.ccs.ornl.gov kernel: [1914704.139462] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8806084c3000 x1471904518249244/t0(0) o38->atlas2-MDT0000-lwp-OST0394@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4e1.ccs.ornl.gov kernel: [1916100.436887] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88093943c800 x1471904515144912/t0(0) o38->atlas2-MDT0000-lwp-OST03c8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4g8.ccs.ornl.gov kernel: [1916101.677557] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8803d3793800 x1471904521746984/t0(0) o38->atlas2-MDT0000-lwp-OST03df@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3a1.ccs.ornl.gov kernel: [1916097.412753] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880a59dc8400 x1471904525931920/t0(0) o38->atlas2-MDT0000-lwp-OST0360@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4d4.ccs.ornl.gov kernel: [1916122.933631] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8804c9d90000 x1471904513679312/t0(0) o38->atlas2-MDT0000-lwp-OST03c3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3f7.ccs.ornl.gov kernel: [802433.658245] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8807347f7c00 x1473076863306348/t0(0) o38->atlas2-MDT0000-lwp-OST038e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3h7.ccs.ornl.gov kernel: [1916109.309953] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8803b3590400 x1471904510333100/t0(0) o38->atlas2-MDT0000-lwp-OST039e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3h3.ccs.ornl.gov kernel: [1916098.182169] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88058308c000 x1471904512413764/t0(0) o38->atlas2-MDT0000-lwp-OST039a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4e2.ccs.ornl.gov kernel: [1916116.402541] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880e087e9800 x1471904514146500/t0(0) o38->atlas2-MDT0000-lwp-OST0339@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4e2.ccs.ornl.gov kernel: [1916116.491796] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss4h8.ccs.ornl.gov kernel: [1916093.536703] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8801cf4b1000 x1471904514267360/t0(0) o38->atlas2-MDT0000-lwp-OST03e7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4b6.ccs.ornl.gov kernel: [1916100.219572] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8801e5663c00 x1471904511213724/t0(0) o38->atlas2-MDT0000-lwp-OST03b5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4b1.ccs.ornl.gov kernel: [1916107.107374] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88070ed3c000 x1471904507949248/t0(0) o38->atlas2-MDT0000-lwp-OST03b0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3g7.ccs.ornl.gov kernel: [1914700.169152] Lustre: 13449:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8802460ea000 x1471904520422408/t0(0) o38->atlas2-MDT0000-lwp-OST0396@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3g7.ccs.ornl.gov kernel: [1914700.261097] Lustre: 13449:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 17 17:09:07 atlas-oss3b3.ccs.ornl.gov kernel: [1916095.581868] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880348c47c00 x1471904513820584/t0(0) o38->atlas2-MDT0000-lwp-OST036a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4g3.ccs.ornl.gov kernel: [1916113.789251] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880c2b34f000 x1471904517610388/t0(0) o38->atlas2-MDT0000-lwp-OST03da@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3a6.ccs.ornl.gov kernel: [1916105.280251] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8801024e8400 x1471904512394800/t0(0) o38->atlas2-MDT0000-lwp-OST0365@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3a5.ccs.ornl.gov kernel: [1916096.176158] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880406b60000 x1471904511738624/t0(0) o38->atlas2-MDT0000-lwp-OST02d4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3a5.ccs.ornl.gov kernel: [1916096.266970] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss3b5.ccs.ornl.gov kernel: [1916115.695488] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8803f2266800 x1471904514717124/t0(0) o38->atlas2-MDT0000-lwp-OST036c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4f6.ccs.ornl.gov kernel: [1916097.458170] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8810251c5c00 x1471904511708716/t0(0) o38->atlas2-MDT0000-lwp-OST0345@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3i4.ccs.ornl.gov kernel: [1916108.553336] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8806978e5400 x1471904510470536/t0(0) o38->atlas2-MDT0000-lwp-OST03a3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4b7.ccs.ornl.gov kernel: [1916096.761032] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880ef9a0dc00 x1471904509377448/t0(0) o38->atlas2-MDT0000-lwp-OST03b6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3h2.ccs.ornl.gov kernel: [1916105.741683] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880ffddd2400 x1471904518760664/t0(0) o38->atlas2-MDT0000-lwp-OST0399@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4f3.ccs.ornl.gov kernel: [1916097.425336] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8801a0169c00 x1471904513977304/t0(0) o38->atlas2-MDT0000-lwp-OST03d2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3h4.ccs.ornl.gov kernel: [1916106.481163] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880bb1529000 x1471904509245420/t0(0) o38->atlas2-MDT0000-lwp-OST039b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4b8.ccs.ornl.gov kernel: [1916104.564506] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880ee3e50800 x1471904512564068/t0(0) o38->atlas2-MDT0000-lwp-OST03b7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3e7.ccs.ornl.gov kernel: [1916105.110001] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff881072904800 x1471904508195368/t0(0) o38->atlas2-MDT0000-lwp-OST0386@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3g1.ccs.ornl.gov kernel: [1914692.678022] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8802967d0c00 x1471904510299788/t0(0) o38->atlas2-MDT0000-lwp-OST0390@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3g2.ccs.ornl.gov kernel: [1914699.357906] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8803b0723400 x1471904509036280/t0(0) o38->atlas2-MDT0000-lwp-OST0391@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3a2.ccs.ornl.gov kernel: [1916102.256083] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880aff61f800 x1471904510524440/t0(0) o38->atlas2-MDT0000-lwp-OST0361@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3h1.ccs.ornl.gov kernel: [1916091.441240] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8801522ecc00 x1471904517766056/t0(0) o38->atlas2-MDT0000-lwp-OST0398@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4f1.ccs.ornl.gov kernel: [1916106.921687] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88042cb97800 x1471904513371220/t0(0) o38->atlas2-MDT0000-lwp-OST03d0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4e3.ccs.ornl.gov kernel: [1916108.559485] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88021afa0000 x1471904515523876/t0(0) o38->atlas2-MDT0000-lwp-OST03ca@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4e8.ccs.ornl.gov kernel: [1916115.697839] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8802518a9400 x1471904518472952/t0(0) o38->atlas2-MDT0000-lwp-OST033f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4e8.ccs.ornl.gov kernel: [1916115.792891] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss4d5.ccs.ornl.gov kernel: [1916115.256398] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8806f5442000 x1471904519136980/t0(0) o38->atlas2-MDT0000-lwp-OST03c4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4i8.ccs.ornl.gov kernel: [1916116.445399] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880606c6f400 x1471904520247284/t0(0) o38->atlas2-MDT0000-lwp-OST03ef@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4i5.ccs.ornl.gov kernel: [1916110.708332] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8802435e1800 x1471904512720564/t0(0) o38->atlas2-MDT0000-lwp-OST03ec@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3i8.ccs.ornl.gov kernel: [1916116.653480] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88086b719000 x1471904515355628/t0(0) o38->atlas2-MDT0000-lwp-OST03a7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3a2.ccs.ornl.gov kernel: [1916102.346881] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss3d8.ccs.ornl.gov kernel: [1916109.321479] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880da542d400 x1471904508710828/t0(0) o38->atlas2-MDT0000-lwp-OST02ef@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3d8.ccs.ornl.gov kernel: [1916109.411178] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss4h5.ccs.ornl.gov kernel: [1916105.270151] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88060ea12400 x1471904509313620/t0(0) o38->atlas2-MDT0000-lwp-OST03e4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3a4.ccs.ornl.gov kernel: [1916092.755614] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8801cd8dcc00 x1471904512538940/t0(0) o38->atlas2-MDT0000-lwp-OST0363@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4b5.ccs.ornl.gov kernel: [1916120.343416] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88062a415c00 x1471904518365816/t0(0) o38->atlas2-MDT0000-lwp-OST03b4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4i6.ccs.ornl.gov kernel: [1916116.167711] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8807085a7400 x1471904510597736/t0(0) o38->atlas2-MDT0000-lwp-OST035d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4h1.ccs.ornl.gov kernel: [1916096.667334] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880c59eb8800 x1471904511603280/t0(0) o38->atlas2-MDT0000-lwp-OST0350@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3d1.ccs.ornl.gov kernel: [1916095.453059] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8801ab929400 x1471904515778468/t0(0) o38->atlas2-MDT0000-lwp-OST02e8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3b6.ccs.ornl.gov kernel: [1916101.020923] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8802f1a73c00 x1471904512275536/t0(0) o38->atlas2-MDT0000-lwp-OST036d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4e5.ccs.ornl.gov kernel: [1916103.881050] Lustre: 24644:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880294584400 x1471904515279364/t0(0) o38->atlas2-MDT0000-lwp-OST03cc@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4e5.ccs.ornl.gov kernel: [1916103.968098] Lustre: 24644:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 17 17:09:07 atlas-oss3b2.ccs.ornl.gov kernel: [1916102.795808] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88104cc5b000 x1471904511917160/t0(0) o38->atlas2-MDT0000-lwp-OST0369@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3e8.ccs.ornl.gov kernel: [1916102.911194] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880180f34400 x1471904515139740/t0(0) o38->atlas2-MDT0000-lwp-OST0387@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4f4.ccs.ornl.gov kernel: [1916103.250018] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880316588c00 x1471904516980416/t0(0) o38->atlas2-MDT0000-lwp-OST0343@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4a6.ccs.ornl.gov kernel: [1916104.871298] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8804068af800 x1471904512502928/t0(0) o38->atlas2-MDT0000-lwp-OST03ad@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3a3.ccs.ornl.gov kernel: [1916100.033506] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880da3c31400 x1471904526380072/t0(0) o38->atlas2-MDT0000-lwp-OST0362@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3e5.ccs.ornl.gov kernel: [1916104.051182] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880a72f74800 x1471904510654960/t0(0) o38->atlas2-MDT0000-lwp-OST0384@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4g1.ccs.ornl.gov kernel: [1916099.895022] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880587e5b000 x1471904508970552/t0(0) o38->atlas2-MDT0000-lwp-OST03d8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3c2.ccs.ornl.gov kernel: [1916104.867314] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88083e453c00 x1471904511648092/t0(0) o38->atlas2-MDT0000-lwp-OST0371@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3g4.ccs.ornl.gov kernel: [1914713.316004] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880a8aa6fc00 x1471904510791520/t0(0) o38->atlas2-MDT0000-lwp-OST0393@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3g4.ccs.ornl.gov kernel: [1914713.405488] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss3c5.ccs.ornl.gov kernel: [1916109.428074] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880e3536c400 x1471904512466280/t0(0) o38->atlas2-MDT0000-lwp-OST0374@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4d6.ccs.ornl.gov kernel: [1916102.914398] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880106cdc400 x1471904519610572/t0(0) o38->atlas2-MDT0000-lwp-OST0335@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3c6.ccs.ornl.gov kernel: [1916095.086800] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88103cfb5400 x1471904508187032/t0(0) o38->atlas2-MDT0000-lwp-OST0375@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3c7.ccs.ornl.gov kernel: [1916103.705668] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880548056c00 x1471904510248832/t0(0) o38->atlas2-MDT0000-lwp-OST02e6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3e4.ccs.ornl.gov kernel: [1916117.090237] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8806e31a3c00 x1471904508538180/t0(0) o38->atlas2-MDT0000-lwp-OST0383@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3d3.ccs.ornl.gov kernel: [1916109.672241] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8802368cf000 x1471904510125876/t0(0) o38->atlas2-MDT0000-lwp-OST037a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4a3.ccs.ornl.gov kernel: [1916118.812352] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880816b5e400 x1471904514689048/t0(0) o38->atlas2-MDT0000-lwp-OST03aa@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4a1.ccs.ornl.gov kernel: [1916100.803662] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880b41f49c00 x1471904510345960/t0(0) o38->atlas2-MDT0000-lwp-OST03a8@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4c6.ccs.ornl.gov kernel: [1916103.761195] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880ca6111800 x1471904517937380/t0(0) o38->atlas2-MDT0000-lwp-OST03bd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3c8.ccs.ornl.gov kernel: [1916107.900073] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8810613b8000 x1471904507928200/t0(0) o38->atlas2-MDT0000-lwp-OST0377@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4a7.ccs.ornl.gov kernel: [1916114.515596] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8802b3415800 x1471904510936176/t0(0) o38->atlas2-MDT0000-lwp-OST03ae@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4e6.ccs.ornl.gov kernel: [1916118.617878] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880bbd51f400 x1471904508408276/t0(0) o38->atlas2-MDT0000-lwp-OST03cd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3c7.ccs.ornl.gov kernel: [1916103.795178] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss3h6.ccs.ornl.gov kernel: [1916103.243195] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880ebc3cd400 x1471904512216680/t0(0) o38->atlas2-MDT0000-lwp-OST039d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3d2.ccs.ornl.gov kernel: [1916104.587173] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880f2097ec00 x1471904511214804/t0(0) o38->atlas2-MDT0000-lwp-OST02e9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3d2.ccs.ornl.gov kernel: [1916104.676965] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss3i6.ccs.ornl.gov kernel: [1916101.735428] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8808667e5000 x1471904509634160/t0(0) o38->atlas2-MDT0000-lwp-OST03a5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3c4.ccs.ornl.gov kernel: [1916098.168949] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88055eb4c000 x1471904511833076/t0(0) o38->atlas2-MDT0000-lwp-OST0373@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4d1.ccs.ornl.gov kernel: [1916098.916028] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880ad17bf400 x1471904516811864/t0(0) o38->atlas2-MDT0000-lwp-OST03c0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3i3.ccs.ornl.gov kernel: [1916098.365137] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8809f4f48400 x1471904518169428/t0(0) o38->atlas2-MDT0000-lwp-OST03a2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3g3.ccs.ornl.gov kernel: [1914700.273738] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880860da9000 x1471904516976916/t0(0) o38->atlas2-MDT0000-lwp-OST0302@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3g3.ccs.ornl.gov kernel: [1914700.370936] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss3a7.ccs.ornl.gov kernel: [1916101.021648] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8802b430d400 x1471904518655784/t0(0) o38->atlas2-MDT0000-lwp-OST0366@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3g6.ccs.ornl.gov kernel: [1914711.276030] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880c3fd6a400 x1471904523473984/t0(0) o38->atlas2-MDT0000-lwp-OST0395@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4h3.ccs.ornl.gov kernel: [1916115.596703] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880ffff12000 x1471904509651000/t0(0) o38->atlas2-MDT0000-lwp-OST03e2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3c3.ccs.ornl.gov kernel: [1916095.596399] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8807a9356800 x1471904511051600/t0(0) o38->atlas2-MDT0000-lwp-OST02e2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3d7.ccs.ornl.gov kernel: [1916103.778818] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880c81929400 x1471904512149604/t0(0) o38->atlas2-MDT0000-lwp-OST037e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4a8.ccs.ornl.gov kernel: [1916108.959468] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8808c6c41800 x1471904507750920/t0(0) o38->atlas2-MDT0000-lwp-OST03af@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4f8.ccs.ornl.gov kernel: [1916102.168447] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880563df6400 x1471904511806184/t0(0) o38->atlas2-MDT0000-lwp-OST03d7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3f1.ccs.ornl.gov kernel: [1916106.753087] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8805b844e400 x1471904508988584/t0(0) o38->atlas2-MDT0000-lwp-OST0388@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss4i1.ccs.ornl.gov kernel: [1916103.028869] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8804ce87e400 x1471904513319884/t0(0) o38->atlas2-MDT0000-lwp-OST0358@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3b7.ccs.ornl.gov kernel: [1916107.810294] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8806b4b02000 x1471904519392796/t0(0) o38->atlas2-MDT0000-lwp-OST024e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3b7.ccs.ornl.gov kernel: [1916107.900556] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss3i1.ccs.ornl.gov kernel: [1916103.085234] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8801acbbe000 x1471904519755560/t0(0) o38->atlas2-MDT0000-lwp-OST0310@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3c3.ccs.ornl.gov kernel: [1916095.685249] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss4d8.ccs.ornl.gov kernel: [1916116.003166] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff8806ffb6c800 x1471904516223148/t0(0) o38->atlas2-MDT0000-lwp-OST03c7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3i1.ccs.ornl.gov kernel: [1916103.173031] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:09:07 atlas-oss3b8.ccs.ornl.gov kernel: [1916117.129725] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880e68a02c00 x1471904516406932/t0(0) o38->atlas2-MDT0000-lwp-OST024f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3b8.ccs.ornl.gov kernel: [1916117.219150] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jul 17 17:09:07 atlas-oss3d5.ccs.ornl.gov kernel: [1916106.483830] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff880bce393c00 x1471904509642184/t0(0) o38->atlas2-MDT0000-lwp-OST037c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3b4.ccs.ornl.gov kernel: [1916114.653627] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88098f6b0400 x1471904510548712/t0(0) o38->atlas2-MDT0000-lwp-OST036b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:09:07 atlas-oss3i2.ccs.ornl.gov kernel: [1916109.197438] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631247/real 1405631347] req@ffff88039a1a7c00 x1471904514016416/t0(0) o38->atlas2-MDT0000-lwp-OST03a1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631527 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:10:23 atlas-oss3i5.ccs.ornl.gov kernel: [702487.037491] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405631143/real 1405631143] req@ffff8801a3203c00 x1473175721933476/t0(0) o38->atlas2-MDT0000-lwp-OST0284@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631423 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:10:23 atlas-oss3i5.ccs.ornl.gov kernel: [702487.129962] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:10:23 atlas-oss3a8.ccs.ornl.gov kernel: [702487.971187] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405631143/real 1405631143] req@ffff88074fbf4400 x1473175720195060/t0(0) o38->atlas2-MDT0000-lwp-OST02d7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631423 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:10:23 atlas-oss3a8.ccs.ornl.gov kernel: [702488.061752] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:10:24 atlas-oss3d4.ccs.ornl.gov kernel: [702487.522101] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405631143/real 1405631143] req@ffff88020f5abc00 x1473175723812856/t0(0) o38->atlas2-MDT0000-lwp-OST00ab@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631423 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:10:24 atlas-oss3d4.ccs.ornl.gov kernel: [702487.610423] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 1 previous similar message Jul 17 17:11:37 atlas-oss4g5.ccs.ornl.gov kernel: [1916252.523178] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631396/real 1405631496] req@ffff8805cd61f800 x1471904510456924/t0(0) o38->atlas2-MDT0000-lwp-OST022c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631676 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4g5.ccs.ornl.gov kernel: [1916252.610295] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4g2.ccs.ornl.gov kernel: [1916263.199430] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631396/real 1405631496] req@ffff880ab2d98c00 x1471904509229144/t0(0) o38->atlas2-MDT0000-lwp-OST0079@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631676 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4h7.ccs.ornl.gov kernel: [1916260.718831] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631396/real 1405631496] req@ffff88064f508800 x1471904512615336/t0(0) o38->atlas2-MDT0000-lwp-OST0236@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631676 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4h7.ccs.ornl.gov kernel: [1916260.805454] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4d2.ccs.ornl.gov kernel: [1916257.536714] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631396/real 1405631497] req@ffff8805c0d9b800 x1471904512221132/t0(0) o38->atlas2-MDT0000-lwp-OST0061@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631676 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4i7.ccs.ornl.gov kernel: [1916265.729599] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880b498acc00 x1471904511375292/t0(0) o38->atlas2-MDT0000-lwp-OST023e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4i7.ccs.ornl.gov kernel: [1916265.819550] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4i2.ccs.ornl.gov kernel: [1916275.320574] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631396/real 1405631497] req@ffff8803bbea3c00 x1471904514634704/t0(0) o38->atlas2-MDT0000-lwp-OST0089@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631676 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4h6.ccs.ornl.gov kernel: [1916251.241293] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880c796c0c00 x1471904515532280/t0(0) o38->atlas2-MDT0000-lwp-OST01a5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4i2.ccs.ornl.gov kernel: [1916275.410138] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4h6.ccs.ornl.gov kernel: [1916251.334700] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4b3.ccs.ornl.gov kernel: [1916246.753496] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631396/real 1405631497] req@ffff8805aeaed800 x1471904509180692/t0(0) o38->atlas2-MDT0000-lwp-OST0052@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631676 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4f7.ccs.ornl.gov kernel: [1916251.525593] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631396/real 1405631497] req@ffff8807672e2400 x1471904510455224/t0(0) o38->atlas2-MDT0000-lwp-OST0076@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631676 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4g2.ccs.ornl.gov kernel: [1916263.289641] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4h2.ccs.ornl.gov kernel: [1916273.694792] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631396/real 1405631497] req@ffff8803200e1000 x1471904508411308/t0(0) o38->atlas2-MDT0000-lwp-OST0081@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631676 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4h2.ccs.ornl.gov kernel: [1916273.790673] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4d2.ccs.ornl.gov kernel: [1916257.633056] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4f7.ccs.ornl.gov kernel: [1916251.618532] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4b3.ccs.ornl.gov kernel: [1916246.847907] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4c1.ccs.ornl.gov kernel: [1916258.156643] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880648741c00 x1471904516097672/t0(0) o38->atlas2-MDT0000-lwp-OST0058@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4c1.ccs.ornl.gov kernel: [1916258.247916] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:11:37 atlas-oss4f5.ccs.ornl.gov kernel: [1916250.629292] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880a0b17d800 x1471904509391096/t0(0) o38->atlas2-MDT0000-lwp-OST0074@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4f5.ccs.ornl.gov kernel: [1916250.720926] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:11:37 atlas-oss4g7.ccs.ornl.gov kernel: [1916275.056939] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880f1ebbb400 x1471904508873744/t0(0) o38->atlas2-MDT0000-lwp-OST007e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4f3.ccs.ornl.gov kernel: [1916247.480139] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880844756800 x1471904513977312/t0(0) o38->atlas2-MDT0000-lwp-OST0072@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4f3.ccs.ornl.gov kernel: [1916247.567951] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4g6.ccs.ornl.gov kernel: [1916262.432974] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff8803b1058c00 x1471904507898860/t0(0) o38->atlas2-MDT0000-lwp-OST010d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4g6.ccs.ornl.gov kernel: [1916262.526224] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:11:37 atlas-oss4g4.ccs.ornl.gov kernel: [1916267.200133] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff88032d6d6000 x1471904515530836/t0(0) o38->atlas2-MDT0000-lwp-OST007b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3g1.ccs.ornl.gov kernel: [1914842.733042] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff88040ca1f800 x1471904510299796/t0(0) o38->atlas2-MDT0000-lwp-OST0030@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3g1.ccs.ornl.gov kernel: [1914842.825465] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss3g2.ccs.ornl.gov kernel: [1914849.412454] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff88013bb54000 x1471904509036288/t0(0) o38->atlas2-MDT0000-lwp-OST0031@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3g2.ccs.ornl.gov kernel: [1914849.502150] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss3a2.ccs.ornl.gov kernel: [1916252.311512] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880a752c9400 x1471904510524452/t0(0) o38->atlas2-MDT0000-lwp-OST0091@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3a2.ccs.ornl.gov kernel: [1916252.402956] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4g7.ccs.ornl.gov kernel: [1916275.145465] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:11:37 atlas-oss4d5.ccs.ornl.gov kernel: [1916265.311604] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880755ce2400 x1471904519136988/t0(0) o38->atlas2-MDT0000-lwp-OST0064@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4d5.ccs.ornl.gov kernel: [1916265.397477] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4g4.ccs.ornl.gov kernel: [1916267.300998] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:11:37 atlas-oss4e3.ccs.ornl.gov kernel: [1916258.615172] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880126aae400 x1471904515523884/t0(0) o38->atlas2-MDT0000-lwp-OST006a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4i5.ccs.ornl.gov kernel: [1916260.764295] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff8804a47abc00 x1471904512720572/t0(0) o38->atlas2-MDT0000-lwp-OST008c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4i5.ccs.ornl.gov kernel: [1916260.853194] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss3i8.ccs.ornl.gov kernel: [1916266.709041] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880739009c00 x1471904515355636/t0(0) o38->atlas2-MDT0000-lwp-OST0047@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4h5.ccs.ornl.gov kernel: [1916255.326086] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff8803d9e68000 x1471904509313628/t0(0) o38->atlas2-MDT0000-lwp-OST0084@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3a4.ccs.ornl.gov kernel: [1916242.810379] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880b49cf2800 x1471904512538948/t0(0) o38->atlas2-MDT0000-lwp-OST0003@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3a4.ccs.ornl.gov kernel: [1916242.896314] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss3d1.ccs.ornl.gov kernel: [1916245.508018] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff88004d28cc00 x1471904515778476/t0(0) o38->atlas2-MDT0000-lwp-OST0018@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4e3.ccs.ornl.gov kernel: [1916258.703989] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss3i8.ccs.ornl.gov kernel: [1916266.800578] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4h1.ccs.ornl.gov kernel: [1916246.722242] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff8808dd8eac00 x1471904511603288/t0(0) o38->atlas2-MDT0000-lwp-OST0080@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3b6.ccs.ornl.gov kernel: [1916251.076207] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff8804fe5e5000 x1471904512275544/t0(0) o38->atlas2-MDT0000-lwp-OST000d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3b6.ccs.ornl.gov kernel: [1916251.164494] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss3b2.ccs.ornl.gov kernel: [1916252.851311] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff88093dc67400 x1471904511917180/t0(0) o38->atlas2-MDT0000-lwp-OST01b9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3e8.ccs.ornl.gov kernel: [1916252.966448] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880452783400 x1471904515139748/t0(0) o38->atlas2-MDT0000-lwp-OST0027@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3b2.ccs.ornl.gov kernel: [1916252.936341] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss3e8.ccs.ornl.gov kernel: [1916253.058040] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4h5.ccs.ornl.gov kernel: [1916255.415422] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss3a3.ccs.ornl.gov kernel: [1916250.088499] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880dfc8a4000 x1471904526380080/t0(0) o38->atlas2-MDT0000-lwp-OST0002@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3a3.ccs.ornl.gov kernel: [1916250.181870] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss3d1.ccs.ornl.gov kernel: [1916245.593736] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss3c2.ccs.ornl.gov kernel: [1916254.922894] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff88016e060000 x1471904511648100/t0(0) o38->atlas2-MDT0000-lwp-OST0011@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3g4.ccs.ornl.gov kernel: [1914863.372729] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880ab6437c00 x1471904510791528/t0(0) o38->atlas2-MDT0000-lwp-OST0033@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3c5.ccs.ornl.gov kernel: [1916259.484139] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880be6240c00 x1471904512466288/t0(0) o38->atlas2-MDT0000-lwp-OST0014@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3c5.ccs.ornl.gov kernel: [1916259.575297] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4h1.ccs.ornl.gov kernel: [1916246.811232] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss3c6.ccs.ornl.gov kernel: [1916245.140803] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880d48857c00 x1471904508187052/t0(0) o38->atlas2-MDT0000-lwp-OST01c5@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3c6.ccs.ornl.gov kernel: [1916245.226908] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss3c2.ccs.ornl.gov kernel: [1916255.009382] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss4c6.ccs.ornl.gov kernel: [1916253.816722] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880e7f1f9c00 x1471904517937388/t0(0) o38->atlas2-MDT0000-lwp-OST005d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4c6.ccs.ornl.gov kernel: [1916253.902228] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss3g4.ccs.ornl.gov kernel: [1914863.462934] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:11:37 atlas-oss3e2.ccs.ornl.gov kernel: [1916252.301841] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff8807770c2000 x1471904513179360/t0(0) o38->atlas2-MDT0000-lwp-OST0021@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3e2.ccs.ornl.gov kernel: [1916252.389712] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Jul 17 17:11:37 atlas-oss3e6.ccs.ornl.gov kernel: [1916250.535193] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880e1ddccc00 x1471904520534800/t0(0) o38->atlas2-MDT0000-lwp-OST0025@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss4b4.ccs.ornl.gov kernel: [1916251.076280] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff880d955f8400 x1471904528872180/t0(0) o38->atlas2-MDT0000-lwp-OST0053@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3b1.ccs.ornl.gov kernel: [1916259.639581] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff8803065fc800 x1471904516110920/t0(0) o38->atlas2-MDT0000-lwp-OST0128@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3b1.ccs.ornl.gov kernel: [1916259.728675] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:11:37 atlas-oss3e6.ccs.ornl.gov kernel: [1916250.625951] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:11:37 atlas-oss4b4.ccs.ornl.gov kernel: [1916251.169757] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:11:37 atlas-oss3i2.ccs.ornl.gov kernel: [1916259.252423] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631397/real 1405631497] req@ffff8801891f6c00 x1471904514016436/t0(0) o38->atlas2-MDT0000-lwp-OST01f1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631677 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:11:37 atlas-oss3i2.ccs.ornl.gov kernel: [1916259.342662] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Jul 17 17:14:07 atlas-oss4e4.ccs.ornl.gov kernel: [1916411.171586] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631546/real 1405631646] req@ffff8806bdeda400 x1471904515374528/t0(0) o38->atlas2-MDT0000-lwp-OST006b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631826 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4e4.ccs.ornl.gov kernel: [1916411.263707] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:14:07 atlas-oss3e3.ccs.ornl.gov kernel: [1916399.232392] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8807761cb000 x1471904510468652/t0(0) o38->atlas2-MDT0000-lwp-OST0022@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4i3.ccs.ornl.gov kernel: [1916394.795460] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880afffadc00 x1471904523159784/t0(0) o38->atlas2-MDT0000-lwp-OST008a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4d7.ccs.ornl.gov kernel: [1916403.272160] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880e57f18800 x1471904514720184/t0(0) o38->atlas2-MDT0000-lwp-OST0066@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4i4.ccs.ornl.gov kernel: [1916407.486762] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880beacee800 x1471904516513632/t0(0) o38->atlas2-MDT0000-lwp-OST008b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4i4.ccs.ornl.gov kernel: [1916407.573134] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3h5.ccs.ornl.gov kernel: [1916422.290582] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8804029c0400 x1471904507806788/t0(0) o38->atlas2-MDT0000-lwp-OST003c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3h5.ccs.ornl.gov kernel: [1916422.383840] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3f6.ccs.ornl.gov kernel: [803107.167245] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880b071ea400 x1473076856558784/t0(0) o38->atlas2-MDT0000-lwp-OST002d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3f6.ccs.ornl.gov kernel: [803107.258512] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3e3.ccs.ornl.gov kernel: [1916399.323090] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3c1.ccs.ornl.gov kernel: [1916404.718601] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8807f90d8000 x1471904511393172/t0(0) o38->atlas2-MDT0000-lwp-OST0010@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3i7.ccs.ornl.gov kernel: [1916399.738391] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88102cae4400 x1471904508389496/t0(0) o38->atlas2-MDT0000-lwp-OST0046@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3i7.ccs.ornl.gov kernel: [1916399.829906] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:07 atlas-oss4f2.ccs.ornl.gov kernel: [1916406.162149] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880f348d2400 x1471904508910884/t0(0) o38->atlas2-MDT0000-lwp-OST0071@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4c4.ccs.ornl.gov kernel: [1916414.470593] Lustre: 25786:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8809f76e3c00 x1471904510981260/t0(0) o38->atlas2-MDT0000-lwp-OST005b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4c4.ccs.ornl.gov kernel: [1916414.560095] Lustre: 25786:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:14:07 atlas-oss4d3.ccs.ornl.gov kernel: [1916399.068231] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880c2c441400 x1471904516978464/t0(0) o38->atlas2-MDT0000-lwp-OST0062@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4d3.ccs.ornl.gov kernel: [1916399.160655] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:07 atlas-oss4i3.ccs.ornl.gov kernel: [1916394.886398] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:07 atlas-oss4d7.ccs.ornl.gov kernel: [1916403.365477] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4h4.ccs.ornl.gov kernel: [1916422.021328] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880357372800 x1471904512334560/t0(0) o38->atlas2-MDT0000-lwp-OST0083@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4h4.ccs.ornl.gov kernel: [1916422.111678] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4c3.ccs.ornl.gov kernel: [1916399.391329] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8809c0309800 x1471904512246276/t0(0) o38->atlas2-MDT0000-lwp-OST005a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4c3.ccs.ornl.gov kernel: [1916399.477461] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4c5.ccs.ornl.gov kernel: [1916398.351380] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88058395d000 x1471904510171424/t0(0) o38->atlas2-MDT0000-lwp-OST005c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3d6.ccs.ornl.gov kernel: [1916417.015457] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880e8fe88800 x1471904511183580/t0(0) o38->atlas2-MDT0000-lwp-OST001d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3d6.ccs.ornl.gov kernel: [1916417.100039] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3g8.ccs.ornl.gov kernel: [1915000.669208] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88086fd03800 x1471904513602616/t0(0) o38->atlas2-MDT0000-lwp-OST0037@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3g8.ccs.ornl.gov kernel: [1915000.755869] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3h8.ccs.ornl.gov kernel: [1916399.487361] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88038a991000 x1471904514773020/t0(0) o38->atlas2-MDT0000-lwp-OST003f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3f4.ccs.ornl.gov kernel: [1916404.794131] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880d431c1c00 x1471904507778676/t0(0) o38->atlas2-MDT0000-lwp-OST002b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4a4.ccs.ornl.gov kernel: [1916404.176733] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880726c8e000 x1471904509591296/t0(0) o38->atlas2-MDT0000-lwp-OST004b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4a4.ccs.ornl.gov kernel: [1916404.265698] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4f2.ccs.ornl.gov kernel: [1916406.253934] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4a5.ccs.ornl.gov kernel: [1916421.603555] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880ac7e02c00 x1471904519454344/t0(0) o38->atlas2-MDT0000-lwp-OST004c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3c1.ccs.ornl.gov kernel: [1916404.811288] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4c8.ccs.ornl.gov kernel: [1916394.505400] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8804ead1d800 x1471904512197116/t0(0) o38->atlas2-MDT0000-lwp-OST005f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4c5.ccs.ornl.gov kernel: [1916398.445062] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3e1.ccs.ornl.gov kernel: [1916405.391909] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880287196c00 x1471904509694544/t0(0) o38->atlas2-MDT0000-lwp-OST0020@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4a2.ccs.ornl.gov kernel: [1916417.281521] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8801aecd3400 x1471904510895908/t0(0) o38->atlas2-MDT0000-lwp-OST0049@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4e7.ccs.ornl.gov kernel: [1916400.054196] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8808223b1400 x1471904522621712/t0(0) o38->atlas2-MDT0000-lwp-OST006e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4e7.ccs.ornl.gov kernel: [1916400.143035] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:07 atlas-oss3f5.ccs.ornl.gov kernel: [803094.543853] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880e53607800 x1473076856406492/t0(0) o38->atlas2-MDT0000-lwp-OST002c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3f5.ccs.ornl.gov kernel: [803094.633790] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:07 atlas-oss3f8.ccs.ornl.gov kernel: [803105.050176] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880ee7e18400 x1473076861321140/t0(0) o38->atlas2-MDT0000-lwp-OST002f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3f8.ccs.ornl.gov kernel: [803105.143525] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3h8.ccs.ornl.gov kernel: [1916399.575730] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3f4.ccs.ornl.gov kernel: [1916404.882893] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:14:07 atlas-oss4c2.ccs.ornl.gov kernel: [1916406.596821] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880ef2ecf000 x1471904508570864/t0(0) o38->atlas2-MDT0000-lwp-OST0059@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4c2.ccs.ornl.gov kernel: [1916406.684938] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3g5.ccs.ornl.gov kernel: [1915004.251138] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88059b1dc800 x1471904518249284/t0(0) o38->atlas2-MDT0000-lwp-OST0034@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3g5.ccs.ornl.gov kernel: [1915004.342723] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4a5.ccs.ornl.gov kernel: [1916421.688585] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4c8.ccs.ornl.gov kernel: [1916394.590197] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3e1.ccs.ornl.gov kernel: [1916405.481481] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4a2.ccs.ornl.gov kernel: [1916417.366535] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4e1.ccs.ornl.gov kernel: [1916400.546926] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff881068f34800 x1471904515144952/t0(0) o38->atlas2-MDT0000-lwp-OST0068@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4e1.ccs.ornl.gov kernel: [1916400.636280] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3h3.ccs.ornl.gov kernel: [1916398.292668] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880304a16000 x1471904512413804/t0(0) o38->atlas2-MDT0000-lwp-OST003a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3h3.ccs.ornl.gov kernel: [1916398.377880] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4g8.ccs.ornl.gov kernel: [1916401.788439] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880543ea0c00 x1471904521747024/t0(0) o38->atlas2-MDT0000-lwp-OST007f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3a1.ccs.ornl.gov kernel: [1916397.522862] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88103c1dcc00 x1471904525931960/t0(0) o38->atlas2-MDT0000-lwp-OST0000@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3a1.ccs.ornl.gov kernel: [1916397.609684] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3f7.ccs.ornl.gov kernel: [802733.770251] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880748dae400 x1473076863306388/t0(0) o38->atlas2-MDT0000-lwp-OST002e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3f7.ccs.ornl.gov kernel: [802733.858449] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4d4.ccs.ornl.gov kernel: [1916423.047532] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88055d495400 x1471904513679352/t0(0) o38->atlas2-MDT0000-lwp-OST0063@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3h7.ccs.ornl.gov kernel: [1916409.420810] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8801cd65e000 x1471904510333140/t0(0) o38->atlas2-MDT0000-lwp-OST003e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3h7.ccs.ornl.gov kernel: [1916409.512306] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4e2.ccs.ornl.gov kernel: [1916416.515625] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880a27288000 x1471904514146544/t0(0) o38->atlas2-MDT0000-lwp-OST03c9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4h8.ccs.ornl.gov kernel: [1916393.646967] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88077f5cbc00 x1471904514267400/t0(0) o38->atlas2-MDT0000-lwp-OST0087@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4h8.ccs.ornl.gov kernel: [1916393.737600] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4b6.ccs.ornl.gov kernel: [1916400.329502] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8801a0833000 x1471904511213764/t0(0) o38->atlas2-MDT0000-lwp-OST0055@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4b6.ccs.ornl.gov kernel: [1916400.421591] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3b3.ccs.ornl.gov kernel: [1916395.691718] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8802ad541000 x1471904513820624/t0(0) o38->atlas2-MDT0000-lwp-OST000a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4b1.ccs.ornl.gov kernel: [1916407.219000] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880406a12c00 x1471904507949288/t0(0) o38->atlas2-MDT0000-lwp-OST0050@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4b1.ccs.ornl.gov kernel: [1916407.307798] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4e2.ccs.ornl.gov kernel: [1916416.606421] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:07 atlas-oss4g8.ccs.ornl.gov kernel: [1916401.877479] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4d4.ccs.ornl.gov kernel: [1916423.139487] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4g3.ccs.ornl.gov kernel: [1916413.901234] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880cac91d400 x1471904517610428/t0(0) o38->atlas2-MDT0000-lwp-OST007a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3b3.ccs.ornl.gov kernel: [1916395.778428] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3a6.ccs.ornl.gov kernel: [1916405.391535] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88036d898c00 x1471904512394840/t0(0) o38->atlas2-MDT0000-lwp-OST0005@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3a5.ccs.ornl.gov kernel: [1916396.286654] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880350017000 x1471904511738664/t0(0) o38->atlas2-MDT0000-lwp-OST0004@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3b5.ccs.ornl.gov kernel: [1916415.808375] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8805522d2c00 x1471904514717164/t0(0) o38->atlas2-MDT0000-lwp-OST000c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3b5.ccs.ornl.gov kernel: [1916415.893058] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4g3.ccs.ornl.gov kernel: [1916413.990989] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4f6.ccs.ornl.gov kernel: [1916397.568814] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff881009c76400 x1471904511708756/t0(0) o38->atlas2-MDT0000-lwp-OST0075@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4f6.ccs.ornl.gov kernel: [1916397.657041] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3i4.ccs.ornl.gov kernel: [1916408.664976] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8805166ea400 x1471904510470576/t0(0) o38->atlas2-MDT0000-lwp-OST0043@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3a6.ccs.ornl.gov kernel: [1916405.479681] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3a5.ccs.ornl.gov kernel: [1916396.379969] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:07 atlas-oss3h2.ccs.ornl.gov kernel: [1916405.853052] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880b36757000 x1471904518760704/t0(0) o38->atlas2-MDT0000-lwp-OST0039@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4b7.ccs.ornl.gov kernel: [1916396.871504] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880c478a9c00 x1471904509377488/t0(0) o38->atlas2-MDT0000-lwp-OST0056@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4b7.ccs.ornl.gov kernel: [1916396.960906] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3h4.ccs.ornl.gov kernel: [1916406.592657] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880154b8c400 x1471904509245460/t0(0) o38->atlas2-MDT0000-lwp-OST003b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4b8.ccs.ornl.gov kernel: [1916404.675763] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880a986f1000 x1471904512564108/t0(0) o38->atlas2-MDT0000-lwp-OST0057@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4b8.ccs.ornl.gov kernel: [1916404.764371] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3e7.ccs.ornl.gov kernel: [1916405.221095] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8808ee74e800 x1471904508195408/t0(0) o38->atlas2-MDT0000-lwp-OST0026@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3i4.ccs.ornl.gov kernel: [1916408.756789] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3h2.ccs.ornl.gov kernel: [1916405.943662] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4e8.ccs.ornl.gov kernel: [1916415.810199] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8802d49e9000 x1471904518472996/t0(0) o38->atlas2-MDT0000-lwp-OST03cf@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4e8.ccs.ornl.gov kernel: [1916415.895836] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:07 atlas-oss3h1.ccs.ornl.gov kernel: [1916391.549325] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88053039b000 x1471904517766096/t0(0) o38->atlas2-MDT0000-lwp-OST0038@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3h4.ccs.ornl.gov kernel: [1916406.686140] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3h1.ccs.ornl.gov kernel: [1916391.641253] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4i8.ccs.ornl.gov kernel: [1916416.556800] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8804a2660000 x1471904520247324/t0(0) o38->atlas2-MDT0000-lwp-OST008f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4i8.ccs.ornl.gov kernel: [1916416.642025] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4f1.ccs.ornl.gov kernel: [1916407.033111] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880366455800 x1471904513371260/t0(0) o38->atlas2-MDT0000-lwp-OST0070@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4f1.ccs.ornl.gov kernel: [1916407.119868] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3e7.ccs.ornl.gov kernel: [1916405.314202] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3f2.ccs.ornl.gov kernel: [1916398.012896] Lustre: 25710:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880f5bd02800 x1471904513672720/t0(0) o38->atlas2-MDT0000-lwp-OST0029@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3f2.ccs.ornl.gov kernel: [1916398.101253] Lustre: 25710:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:14:07 atlas-oss4b5.ccs.ornl.gov kernel: [1916420.457113] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88023944c800 x1471904518365856/t0(0) o38->atlas2-MDT0000-lwp-OST0054@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4b5.ccs.ornl.gov kernel: [1916420.541903] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3d8.ccs.ornl.gov kernel: [1916409.433805] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880cdf811800 x1471904508710868/t0(0) o38->atlas2-MDT0000-lwp-OST001f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3d8.ccs.ornl.gov kernel: [1916409.535400] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:07 atlas-oss4i6.ccs.ornl.gov kernel: [1916416.280420] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8803abfc9400 x1471904510597776/t0(0) o38->atlas2-MDT0000-lwp-OST008d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4i6.ccs.ornl.gov kernel: [1916416.373520] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4f4.ccs.ornl.gov kernel: [1916403.360900] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88030a45ac00 x1471904516980456/t0(0) o38->atlas2-MDT0000-lwp-OST0073@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4f4.ccs.ornl.gov kernel: [1916403.445954] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4a6.ccs.ornl.gov kernel: [1916404.982609] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8806db25b000 x1471904512502968/t0(0) o38->atlas2-MDT0000-lwp-OST004d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4a6.ccs.ornl.gov kernel: [1916405.069643] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3e5.ccs.ornl.gov kernel: [1916404.161912] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880ad91c0c00 x1471904510655000/t0(0) o38->atlas2-MDT0000-lwp-OST0024@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4g1.ccs.ornl.gov kernel: [1916400.005687] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8801d02c0400 x1471904508970592/t0(0) o38->atlas2-MDT0000-lwp-OST0078@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4g1.ccs.ornl.gov kernel: [1916400.092174] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4d6.ccs.ornl.gov kernel: [1916403.012458] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88081eeb0400 x1471904519610612/t0(0) o38->atlas2-MDT0000-lwp-OST0065@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3f3.ccs.ornl.gov kernel: [1916416.466671] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88051024d800 x1471904510688284/t0(0) o38->atlas2-MDT0000-lwp-OST002a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3f3.ccs.ornl.gov kernel: [1916416.555998] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:14:07 atlas-oss3c7.ccs.ornl.gov kernel: [1916403.816730] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880386ef3800 x1471904510248872/t0(0) o38->atlas2-MDT0000-lwp-OST0016@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3e4.ccs.ornl.gov kernel: [1916417.203556] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880826e46c00 x1471904508538220/t0(0) o38->atlas2-MDT0000-lwp-OST0023@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4a3.ccs.ornl.gov kernel: [1916418.925820] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8804f592e000 x1471904514689088/t0(0) o38->atlas2-MDT0000-lwp-OST004a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3e5.ccs.ornl.gov kernel: [1916404.253090] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3d3.ccs.ornl.gov kernel: [1916409.784353] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8804451f5c00 x1471904510125916/t0(0) o38->atlas2-MDT0000-lwp-OST001a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3d3.ccs.ornl.gov kernel: [1916409.875539] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4a1.ccs.ornl.gov kernel: [1916400.914003] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880bef4ef400 x1471904510346000/t0(0) o38->atlas2-MDT0000-lwp-OST0048@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4a1.ccs.ornl.gov kernel: [1916401.001752] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4d6.ccs.ornl.gov kernel: [1916403.104379] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4a7.ccs.ornl.gov kernel: [1916414.628275] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880259af7800 x1471904510936216/t0(0) o38->atlas2-MDT0000-lwp-OST004e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4a7.ccs.ornl.gov kernel: [1916414.721701] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4e6.ccs.ornl.gov kernel: [1916418.731255] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880f0c519400 x1471904508408316/t0(0) o38->atlas2-MDT0000-lwp-OST006d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3c8.ccs.ornl.gov kernel: [1916408.010377] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880e9bd72000 x1471904507928240/t0(0) o38->atlas2-MDT0000-lwp-OST0017@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3h6.ccs.ornl.gov kernel: [1916403.354274] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8808c74cdc00 x1471904512216720/t0(0) o38->atlas2-MDT0000-lwp-OST003d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3h6.ccs.ornl.gov kernel: [1916403.447469] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3c7.ccs.ornl.gov kernel: [1916403.907665] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:07 atlas-oss3d2.ccs.ornl.gov kernel: [1916404.697767] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880992492c00 x1471904511214844/t0(0) o38->atlas2-MDT0000-lwp-OST0019@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3d2.ccs.ornl.gov kernel: [1916404.788980] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:07 atlas-oss3e4.ccs.ornl.gov kernel: [1916417.293626] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3i6.ccs.ornl.gov kernel: [1916401.845751] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8801d7a22000 x1471904509634200/t0(0) o38->atlas2-MDT0000-lwp-OST0045@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3i6.ccs.ornl.gov kernel: [1916401.939046] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3c4.ccs.ornl.gov kernel: [1916398.279309] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8803cda39000 x1471904511833116/t0(0) o38->atlas2-MDT0000-lwp-OST0013@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4a3.ccs.ornl.gov kernel: [1916419.011984] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3g3.ccs.ornl.gov kernel: [1915000.385255] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88069f415800 x1471904516976960/t0(0) o38->atlas2-MDT0000-lwp-OST0392@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4d1.ccs.ornl.gov kernel: [1916399.026471] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880a9702f400 x1471904516811904/t0(0) o38->atlas2-MDT0000-lwp-OST0060@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3a7.ccs.ornl.gov kernel: [1916401.132206] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8805d7cbf400 x1471904518655824/t0(0) o38->atlas2-MDT0000-lwp-OST0006@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4d1.ccs.ornl.gov kernel: [1916399.116162] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4e6.ccs.ornl.gov kernel: [1916418.823219] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3i3.ccs.ornl.gov kernel: [1916398.475305] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880aca22a400 x1471904518169468/t0(0) o38->atlas2-MDT0000-lwp-OST0042@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3i3.ccs.ornl.gov kernel: [1916398.566996] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3c8.ccs.ornl.gov kernel: [1916408.114148] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3g6.ccs.ornl.gov kernel: [1915011.389032] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88102c3dbc00 x1471904523474024/t0(0) o38->atlas2-MDT0000-lwp-OST0035@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4h3.ccs.ornl.gov kernel: [1916415.709119] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880e4e289000 x1471904509651040/t0(0) o38->atlas2-MDT0000-lwp-OST0082@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3c3.ccs.ornl.gov kernel: [1916395.706494] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88062830e800 x1471904511051640/t0(0) o38->atlas2-MDT0000-lwp-OST0012@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3c3.ccs.ornl.gov kernel: [1916395.796898] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:07 atlas-oss3d7.ccs.ornl.gov kernel: [1916403.889863] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880ba412d400 x1471904512149644/t0(0) o38->atlas2-MDT0000-lwp-OST001e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4i1.ccs.ornl.gov kernel: [1916403.138613] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88017c04c400 x1471904513319924/t0(0) o38->atlas2-MDT0000-lwp-OST0088@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4a8.ccs.ornl.gov kernel: [1916409.071222] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880ff9c55800 x1471904507750960/t0(0) o38->atlas2-MDT0000-lwp-OST004f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4f8.ccs.ornl.gov kernel: [1916402.279370] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8801a5bb2800 x1471904511806224/t0(0) o38->atlas2-MDT0000-lwp-OST0077@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3g3.ccs.ornl.gov kernel: [1915000.473765] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:07 atlas-oss3f1.ccs.ornl.gov kernel: [1916406.863622] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880387862000 x1471904508988624/t0(0) o38->atlas2-MDT0000-lwp-OST0028@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3f1.ccs.ornl.gov kernel: [1916406.956723] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3c4.ccs.ornl.gov kernel: [1916398.369834] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3b7.ccs.ornl.gov kernel: [1916407.921570] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88020891ac00 x1471904519392836/t0(0) o38->atlas2-MDT0000-lwp-OST000e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3b7.ccs.ornl.gov kernel: [1916408.013394] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:07 atlas-oss3a7.ccs.ornl.gov kernel: [1916401.217205] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3g6.ccs.ornl.gov kernel: [1915011.481848] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3i1.ccs.ornl.gov kernel: [1916403.196436] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8806b3e3d400 x1471904519755600/t0(0) o38->atlas2-MDT0000-lwp-OST0040@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3i1.ccs.ornl.gov kernel: [1916403.285033] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:07 atlas-oss3d7.ccs.ornl.gov kernel: [1916403.979783] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4a8.ccs.ornl.gov kernel: [1916409.161660] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4d8.ccs.ornl.gov kernel: [1916416.116218] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88047b45c400 x1471904516223188/t0(0) o38->atlas2-MDT0000-lwp-OST0067@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4h3.ccs.ornl.gov kernel: [1916415.799791] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4i1.ccs.ornl.gov kernel: [1916403.229767] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss4f8.ccs.ornl.gov kernel: [1916402.365967] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3b8.ccs.ornl.gov kernel: [1916417.242776] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff88101e512c00 x1471904516406972/t0(0) o38->atlas2-MDT0000-lwp-OST000f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss4d8.ccs.ornl.gov kernel: [1916416.203520] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3d5.ccs.ornl.gov kernel: [1916406.595360] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff880d2216dc00 x1471904509642224/t0(0) o38->atlas2-MDT0000-lwp-OST001c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3b8.ccs.ornl.gov kernel: [1916417.333789] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jul 17 17:14:07 atlas-oss3d5.ccs.ornl.gov kernel: [1916406.687162] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:07 atlas-oss3b4.ccs.ornl.gov kernel: [1916414.766274] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631547/real 1405631647] req@ffff8810021f4400 x1471904510548752/t0(0) o38->atlas2-MDT0000-lwp-OST000b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631827 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:07 atlas-oss3b4.ccs.ornl.gov kernel: [1916414.854519] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:14:53 atlas-oss3i5.ccs.ornl.gov kernel: [702757.138055] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631593/real 1405631693] req@ffff8808709c8c00 x1473175721933528/t0(0) o38->atlas2-MDT0000-lwp-OST00d4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631873 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:53 atlas-oss3i5.ccs.ornl.gov kernel: [702757.231487] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:53 atlas-oss3a8.ccs.ornl.gov kernel: [702758.070815] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631593/real 1405631693] req@ffff8802ce9dd400 x1473175720195112/t0(0) o38->atlas2-MDT0000-lwp-OST0097@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631873 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:53 atlas-oss3a8.ccs.ornl.gov kernel: [702758.163142] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:14:54 atlas-oss3d4.ccs.ornl.gov kernel: [702757.622629] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631593/real 1405631693] req@ffff8802842c6c00 x1473175723812908/t0(0) o38->atlas2-MDT0000-lwp-OST013b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631873 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:14:54 atlas-oss3d4.ccs.ornl.gov kernel: [702757.712301] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Jul 17 17:16:37 atlas-oss4g5.ccs.ornl.gov kernel: [1916552.633447] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631696/real 1405631796] req@ffff880680147000 x1471904510456952/t0(0) o38->atlas2-MDT0000-lwp-OST007c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631976 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4g2.ccs.ornl.gov kernel: [1916563.311959] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631696/real 1405631796] req@ffff880c8b9dd800 x1471904509229184/t0(0) o38->atlas2-MDT0000-lwp-OST0079@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631976 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4h7.ccs.ornl.gov kernel: [1916560.830504] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631696/real 1405631796] req@ffff8801b968a000 x1471904512615376/t0(0) o38->atlas2-MDT0000-lwp-OST0236@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631976 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4d2.ccs.ornl.gov kernel: [1916557.648304] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff8804d641a400 x1471904512221188/t0(0) o38->atlas2-MDT0000-lwp-OST0211@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4d2.ccs.ornl.gov kernel: [1916557.735444] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4h6.ccs.ornl.gov kernel: [1916551.351687] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff880fde555000 x1471904515532316/t0(0) o38->atlas2-MDT0000-lwp-OST0115@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4h6.ccs.ornl.gov kernel: [1916551.437541] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4i7.ccs.ornl.gov kernel: [1916565.842780] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff8802a14cac00 x1471904511375332/t0(0) o38->atlas2-MDT0000-lwp-OST023e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4i7.ccs.ornl.gov kernel: [1916565.934487] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4b3.ccs.ornl.gov kernel: [1916546.863467] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631696/real 1405631797] req@ffff880775db5400 x1471904509180732/t0(0) o38->atlas2-MDT0000-lwp-OST0052@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631976 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4b3.ccs.ornl.gov kernel: [1916546.950060] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4i2.ccs.ornl.gov kernel: [1916575.435101] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631696/real 1405631797] req@ffff88063c638c00 x1471904514634744/t0(0) o38->atlas2-MDT0000-lwp-OST0089@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631976 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4i2.ccs.ornl.gov kernel: [1916575.525927] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4g5.ccs.ornl.gov kernel: [1916552.722086] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4f7.ccs.ornl.gov kernel: [1916551.636360] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631696/real 1405631797] req@ffff88075b3af000 x1471904510455264/t0(0) o38->atlas2-MDT0000-lwp-OST0076@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631976 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4g2.ccs.ornl.gov kernel: [1916563.403196] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4h2.ccs.ornl.gov kernel: [1916573.809932] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631696/real 1405631797] req@ffff88042c6ac400 x1471904508411348/t0(0) o38->atlas2-MDT0000-lwp-OST0081@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631976 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4h2.ccs.ornl.gov kernel: [1916573.895696] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4h7.ccs.ornl.gov kernel: [1916560.917840] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4f7.ccs.ornl.gov kernel: [1916551.721176] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4f3.ccs.ornl.gov kernel: [1916547.589933] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff8804e2af5800 x1471904513977364/t0(0) o38->atlas2-MDT0000-lwp-OST02b2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4f3.ccs.ornl.gov kernel: [1916547.678839] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss3g1.ccs.ornl.gov kernel: [1915142.842974] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff88041ae33c00 x1471904510299840/t0(0) o38->atlas2-MDT0000-lwp-OST00c0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss3a2.ccs.ornl.gov kernel: [1916552.422356] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff88097e426000 x1471904510524488/t0(0) o38->atlas2-MDT0000-lwp-OST0001@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss3a2.ccs.ornl.gov kernel: [1916552.515506] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss3g2.ccs.ornl.gov kernel: [1915149.523599] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff8803b0a89800 x1471904509036328/t0(0) o38->atlas2-MDT0000-lwp-OST0031@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss3g1.ccs.ornl.gov kernel: [1915142.927930] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4d5.ccs.ornl.gov kernel: [1916565.424007] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff8806fa84a000 x1471904519137028/t0(0) o38->atlas2-MDT0000-lwp-OST0064@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4d5.ccs.ornl.gov kernel: [1916565.512461] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4e3.ccs.ornl.gov kernel: [1916558.726608] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff88101093f400 x1471904515523924/t0(0) o38->atlas2-MDT0000-lwp-OST006a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4i5.ccs.ornl.gov kernel: [1916560.876210] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff8806bf237800 x1471904512720612/t0(0) o38->atlas2-MDT0000-lwp-OST008c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss3i8.ccs.ornl.gov kernel: [1916566.822183] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff88074eeb3c00 x1471904515355676/t0(0) o38->atlas2-MDT0000-lwp-OST0047@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss3i8.ccs.ornl.gov kernel: [1916566.915363] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4h5.ccs.ornl.gov kernel: [1916555.437973] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff88060e7db800 x1471904509313668/t0(0) o38->atlas2-MDT0000-lwp-OST0084@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4h5.ccs.ornl.gov kernel: [1916555.528526] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss3a4.ccs.ornl.gov kernel: [1916542.919917] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff88106fb06400 x1471904512538988/t0(0) o38->atlas2-MDT0000-lwp-OST0003@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4c7.ccs.ornl.gov kernel: [1916552.813220] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff88039bf80400 x1471904514425820/t0(0) o38->atlas2-MDT0000-lwp-OST005e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4c7.ccs.ornl.gov kernel: [1916552.898134] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:16:37 atlas-oss3g2.ccs.ornl.gov kernel: [1915149.625233] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4e3.ccs.ornl.gov kernel: [1916558.816511] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4h1.ccs.ornl.gov kernel: [1916546.832017] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff880dfd08f000 x1471904511603328/t0(0) o38->atlas2-MDT0000-lwp-OST0080@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss4h1.ccs.ornl.gov kernel: [1916546.922346] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss3b6.ccs.ornl.gov kernel: [1916551.186792] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff8803ae566c00 x1471904512275584/t0(0) o38->atlas2-MDT0000-lwp-OST000d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss3b6.ccs.ornl.gov kernel: [1916551.276848] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss3d1.ccs.ornl.gov kernel: [1916545.617948] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff8807db10dc00 x1471904515778516/t0(0) o38->atlas2-MDT0000-lwp-OST0018@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss3a3.ccs.ornl.gov kernel: [1916550.198571] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff88102997bc00 x1471904526380132/t0(0) o38->atlas2-MDT0000-lwp-OST01b2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss3a3.ccs.ornl.gov kernel: [1916550.284376] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4i5.ccs.ornl.gov kernel: [1916560.966328] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss3e8.ccs.ornl.gov kernel: [1916553.076957] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff8805643cf000 x1471904515139788/t0(0) o38->atlas2-MDT0000-lwp-OST0027@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss3b2.ccs.ornl.gov kernel: [1916552.962258] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff88095dd49000 x1471904511917208/t0(0) o38->atlas2-MDT0000-lwp-OST0009@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss3a4.ccs.ornl.gov kernel: [1916543.008861] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss3c2.ccs.ornl.gov kernel: [1916555.034066] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff88034b2cb000 x1471904511648140/t0(0) o38->atlas2-MDT0000-lwp-OST0011@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss3c2.ccs.ornl.gov kernel: [1916555.121375] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss3g4.ccs.ornl.gov kernel: [1915163.486297] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff880c1d0e3c00 x1471904510791572/t0(0) o38->atlas2-MDT0000-lwp-OST00c3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss3c5.ccs.ornl.gov kernel: [1916559.596284] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff880fd31d6800 x1471904512466328/t0(0) o38->atlas2-MDT0000-lwp-OST0014@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss3e8.ccs.ornl.gov kernel: [1916553.170703] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss3c6.ccs.ornl.gov kernel: [1916545.250656] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff880ace741800 x1471904508187080/t0(0) o38->atlas2-MDT0000-lwp-OST0015@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss3c6.ccs.ornl.gov kernel: [1916545.337756] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss3d1.ccs.ornl.gov kernel: [1916545.716216] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss3b2.ccs.ornl.gov kernel: [1916553.050317] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4c6.ccs.ornl.gov kernel: [1916553.927895] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff8810002d7800 x1471904517937444/t0(0) o38->atlas2-MDT0000-lwp-OST029d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:37 atlas-oss3c5.ccs.ornl.gov kernel: [1916559.689263] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss3g4.ccs.ornl.gov kernel: [1915163.578078] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss4c6.ccs.ornl.gov kernel: [1916554.015696] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:16:37 atlas-oss3i2.ccs.ornl.gov kernel: [1916559.364261] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631697/real 1405631797] req@ffff8805ce18e400 x1471904514016464/t0(0) o38->atlas2-MDT0000-lwp-OST0041@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405631977 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:16:38 atlas-oss3i2.ccs.ornl.gov kernel: [1916559.456035] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Jul 17 17:21:37 atlas-oss4b2.ccs.ornl.gov kernel: [1916875.519766] Lustre: 27461:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631997/real 1405632097] req@ffff880af12a3400 x1471904518026104/t0(0) o38->atlas2-MDT0000-lwp-OST0171@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632277 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:21:37 atlas-oss4b2.ccs.ornl.gov kernel: [1916875.607435] Lustre: 27461:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:21:37 atlas-oss4c1.ccs.ornl.gov kernel: [1916858.380652] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631997/real 1405632097] req@ffff880440b4a400 x1471904516097764/t0(0) o38->atlas2-MDT0000-lwp-OST0208@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632277 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:21:37 atlas-oss3g7.ccs.ornl.gov kernel: [1915450.446665] Lustre: 13449:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631997/real 1405632097] req@ffff88037a5cc800 x1471904520422508/t0(0) o38->atlas2-MDT0000-lwp-OST01e6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632277 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:21:37 atlas-oss3g7.ccs.ornl.gov kernel: [1915450.535214] Lustre: 13449:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:21:37 atlas-oss4c1.ccs.ornl.gov kernel: [1916858.465391] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:21:37 atlas-oss4f5.ccs.ornl.gov kernel: [1916850.851947] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631997/real 1405632097] req@ffff881048485c00 x1471904509391176/t0(0) o38->atlas2-MDT0000-lwp-OST0074@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632277 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:21:37 atlas-oss4f5.ccs.ornl.gov kernel: [1916850.937277] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:21:37 atlas-oss4g6.ccs.ornl.gov kernel: [1916862.658832] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631997/real 1405632097] req@ffff880871fbfc00 x1471904507898936/t0(0) o38->atlas2-MDT0000-lwp-OST007d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632277 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:21:37 atlas-oss4g4.ccs.ornl.gov kernel: [1916867.426453] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631997/real 1405632097] req@ffff8807bf607800 x1471904515530916/t0(0) o38->atlas2-MDT0000-lwp-OST007b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632277 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:21:37 atlas-oss4g7.ccs.ornl.gov kernel: [1916875.284982] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631997/real 1405632097] req@ffff880aa75a7400 x1471904508873836/t0(0) o38->atlas2-MDT0000-lwp-OST022e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632277 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:21:37 atlas-oss4g6.ccs.ornl.gov kernel: [1916862.747261] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:21:37 atlas-oss4g4.ccs.ornl.gov kernel: [1916867.520146] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:21:37 atlas-oss4g7.ccs.ornl.gov kernel: [1916875.375564] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:21:37 atlas-oss4e5.ccs.ornl.gov kernel: [1916854.159823] Lustre: 24644:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631997/real 1405632097] req@ffff88065e603800 x1471904515279452/t0(0) o38->atlas2-MDT0000-lwp-OST006c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632277 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:21:37 atlas-oss4e5.ccs.ornl.gov kernel: [1916854.249117] Lustre: 24644:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:21:37 atlas-oss3e2.ccs.ornl.gov kernel: [1916852.523320] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631997/real 1405632097] req@ffff880870900800 x1471904513179444/t0(0) o38->atlas2-MDT0000-lwp-OST00b1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632277 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:21:37 atlas-oss3e2.ccs.ornl.gov kernel: [1916852.615272] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:21:37 atlas-oss4b4.ccs.ornl.gov kernel: [1916851.296473] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631997/real 1405632097] req@ffff880cd6c6b000 x1471904528872260/t0(0) o38->atlas2-MDT0000-lwp-OST0053@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632277 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:21:37 atlas-oss4b4.ccs.ornl.gov kernel: [1916851.385747] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:21:37 atlas-oss3e6.ccs.ornl.gov kernel: [1916850.756199] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631997/real 1405632097] req@ffff88096d60f800 x1471904520534880/t0(0) o38->atlas2-MDT0000-lwp-OST0025@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632277 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:21:37 atlas-oss3e6.ccs.ornl.gov kernel: [1916850.849482] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:21:37 atlas-oss3b1.ccs.ornl.gov kernel: [1916859.863844] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405631997/real 1405632097] req@ffff88065bf45800 x1471904516110992/t0(0) o38->atlas2-MDT0000-lwp-OST0008@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632277 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:21:37 atlas-oss3b1.ccs.ornl.gov kernel: [1916859.955516] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:23:44 atlas-oss4a2.ccs.ornl.gov kernel: [1916995.228922] LustreError: 81447:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Jul 17 17:23:45 atlas-oss3f7.ccs.ornl.gov kernel: [803311.915582] LustreError: 115844:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Jul 17 17:23:45 atlas-oss4a8.ccs.ornl.gov kernel: [1916987.011377] LustreError: 42367:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Jul 17 17:23:45 atlas-oss3b1.ccs.ornl.gov kernel: [1916987.698819] LustreError: 109854:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Jul 17 17:24:07 atlas-oss3e3.ccs.ornl.gov kernel: [1916999.453120] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880342cdec00 x1471904510468732/t0(0) o38->atlas2-MDT0000-lwp-OST0022@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3e3.ccs.ornl.gov kernel: [1916999.546104] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4d7.ccs.ornl.gov kernel: [1917003.495370] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880a7f89d400 x1471904514720264/t0(0) o38->atlas2-MDT0000-lwp-OST0066@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3h5.ccs.ornl.gov kernel: [1917022.518662] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88055fca7400 x1471904507806868/t0(0) o38->atlas2-MDT0000-lwp-OST003c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3h5.ccs.ornl.gov kernel: [1917022.606511] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3f6.ccs.ornl.gov kernel: [803707.390747] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8808d5be9000 x1473076856558864/t0(0) o38->atlas2-MDT0000-lwp-OST002d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3f6.ccs.ornl.gov kernel: [803707.475987] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4i3.ccs.ornl.gov kernel: [1916995.014641] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880fd0aa7c00 x1471904523159864/t0(0) o38->atlas2-MDT0000-lwp-OST008a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4i4.ccs.ornl.gov kernel: [1917007.710750] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880d142b4c00 x1471904516513712/t0(0) o38->atlas2-MDT0000-lwp-OST008b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3c1.ccs.ornl.gov kernel: [1917004.941446] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88087068c800 x1471904511393252/t0(0) o38->atlas2-MDT0000-lwp-OST0010@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3c1.ccs.ornl.gov kernel: [1917005.027317] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3i7.ccs.ornl.gov kernel: [1916999.960041] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880d3d19c400 x1471904508389576/t0(0) o38->atlas2-MDT0000-lwp-OST0046@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3i7.ccs.ornl.gov kernel: [1917000.045426] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4f2.ccs.ornl.gov kernel: [1917006.385239] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88093f8d0c00 x1471904508910964/t0(0) o38->atlas2-MDT0000-lwp-OST0071@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4f2.ccs.ornl.gov kernel: [1917006.470175] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4d3.ccs.ornl.gov kernel: [1916999.289144] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880db6780c00 x1471904516978544/t0(0) o38->atlas2-MDT0000-lwp-OST0062@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4d7.ccs.ornl.gov kernel: [1917003.583499] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4c5.ccs.ornl.gov kernel: [1916998.571755] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8805a55e8800 x1471904510171504/t0(0) o38->atlas2-MDT0000-lwp-OST005c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4h4.ccs.ornl.gov kernel: [1917022.248485] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8803d691a800 x1471904512334640/t0(0) o38->atlas2-MDT0000-lwp-OST0083@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4h4.ccs.ornl.gov kernel: [1917022.338681] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4c3.ccs.ornl.gov kernel: [1916999.612111] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88100005a000 x1471904512246356/t0(0) o38->atlas2-MDT0000-lwp-OST005a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4c3.ccs.ornl.gov kernel: [1916999.702995] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4i3.ccs.ornl.gov kernel: [1916995.117884] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3g8.ccs.ornl.gov kernel: [1915600.891759] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8804163f8000 x1471904513602696/t0(0) o38->atlas2-MDT0000-lwp-OST0037@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3g8.ccs.ornl.gov kernel: [1915600.980863] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4i4.ccs.ornl.gov kernel: [1917007.802472] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3d6.ccs.ornl.gov kernel: [1917017.241750] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88103f2da400 x1471904511183660/t0(0) o38->atlas2-MDT0000-lwp-OST001d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3h8.ccs.ornl.gov kernel: [1916999.708623] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880488f23400 x1471904514773100/t0(0) o38->atlas2-MDT0000-lwp-OST003f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4a4.ccs.ornl.gov kernel: [1917004.398907] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880786fb8c00 x1471904509591376/t0(0) o38->atlas2-MDT0000-lwp-OST004b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4d3.ccs.ornl.gov kernel: [1916999.375818] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4a4.ccs.ornl.gov kernel: [1917004.490326] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4a5.ccs.ornl.gov kernel: [1917021.832443] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880dff4de000 x1471904519454424/t0(0) o38->atlas2-MDT0000-lwp-OST004c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4a5.ccs.ornl.gov kernel: [1917021.921675] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4c5.ccs.ornl.gov kernel: [1916998.660487] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3e1.ccs.ornl.gov kernel: [1917005.614788] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88075e8e9c00 x1471904509694624/t0(0) o38->atlas2-MDT0000-lwp-OST0020@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4a2.ccs.ornl.gov kernel: [1917017.507516] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8804805a2000 x1471904510895988/t0(0) o38->atlas2-MDT0000-lwp-OST0049@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4a2.ccs.ornl.gov kernel: [1917017.596615] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4c8.ccs.ornl.gov kernel: [1916994.727075] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88037fb2ac00 x1471904512197196/t0(0) o38->atlas2-MDT0000-lwp-OST005f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4c8.ccs.ornl.gov kernel: [1916994.820352] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4e7.ccs.ornl.gov kernel: [1917000.274191] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880158235c00 x1471904522621796/t0(0) o38->atlas2-MDT0000-lwp-OST03ce@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3f5.ccs.ornl.gov kernel: [803694.763140] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880a3408c800 x1473076856406572/t0(0) o38->atlas2-MDT0000-lwp-OST002c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3f8.ccs.ornl.gov kernel: [803705.271623] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880ccf28dc00 x1473076861321220/t0(0) o38->atlas2-MDT0000-lwp-OST002f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3d6.ccs.ornl.gov kernel: [1917017.330753] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3h8.ccs.ornl.gov kernel: [1916999.799057] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4c2.ccs.ornl.gov kernel: [1917006.818916] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff881072fad400 x1471904508570944/t0(0) o38->atlas2-MDT0000-lwp-OST0059@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4c2.ccs.ornl.gov kernel: [1917006.909592] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3g5.ccs.ornl.gov kernel: [1915604.474579] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88025b347000 x1471904518249364/t0(0) o38->atlas2-MDT0000-lwp-OST0034@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3g5.ccs.ornl.gov kernel: [1915604.567806] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3e1.ccs.ornl.gov kernel: [1917005.706650] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3f5.ccs.ornl.gov kernel: [803694.856006] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3f8.ccs.ornl.gov kernel: [803705.360177] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4e1.ccs.ornl.gov kernel: [1917000.767140] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880d62509c00 x1471904515145032/t0(0) o38->atlas2-MDT0000-lwp-OST0068@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4e7.ccs.ornl.gov kernel: [1917000.366297] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4d4.ccs.ornl.gov kernel: [1917023.275524] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8805cedf5400 x1471904513679432/t0(0) o38->atlas2-MDT0000-lwp-OST0063@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4g8.ccs.ornl.gov kernel: [1917002.010561] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880125409000 x1471904521747104/t0(0) o38->atlas2-MDT0000-lwp-OST007f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4g8.ccs.ornl.gov kernel: [1917002.101215] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3h3.ccs.ornl.gov kernel: [1916998.513564] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88074d58f000 x1471904512413884/t0(0) o38->atlas2-MDT0000-lwp-OST003a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3h3.ccs.ornl.gov kernel: [1916998.603401] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3a1.ccs.ornl.gov kernel: [1916997.743006] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff881014328800 x1471904525932040/t0(0) o38->atlas2-MDT0000-lwp-OST0000@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3f7.ccs.ornl.gov kernel: [803333.994240] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880100988800 x1473076863306468/t0(0) o38->atlas2-MDT0000-lwp-OST002e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3f7.ccs.ornl.gov kernel: [803334.084825] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3h7.ccs.ornl.gov kernel: [1917009.644662] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8803c23d7c00 x1471904510333220/t0(0) o38->atlas2-MDT0000-lwp-OST003e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3h7.ccs.ornl.gov kernel: [1917009.738135] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4e2.ccs.ornl.gov kernel: [1917016.741841] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8809bcbf4000 x1471904514146620/t0(0) o38->atlas2-MDT0000-lwp-OST0069@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4b6.ccs.ornl.gov kernel: [1917000.549521] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8805f244e800 x1471904511213844/t0(0) o38->atlas2-MDT0000-lwp-OST0055@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4h8.ccs.ornl.gov kernel: [1916993.867384] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8807116bf800 x1471904514267480/t0(0) o38->atlas2-MDT0000-lwp-OST0087@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3b3.ccs.ornl.gov kernel: [1916995.911406] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8807aa71a000 x1471904513820704/t0(0) o38->atlas2-MDT0000-lwp-OST000a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4b1.ccs.ornl.gov kernel: [1917007.442278] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88007b075c00 x1471904507949368/t0(0) o38->atlas2-MDT0000-lwp-OST0050@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4b1.ccs.ornl.gov kernel: [1917007.533413] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4d4.ccs.ornl.gov kernel: [1917023.360217] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4e1.ccs.ornl.gov kernel: [1917000.859639] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3a1.ccs.ornl.gov kernel: [1916997.834790] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4e2.ccs.ornl.gov kernel: [1917016.835021] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4b6.ccs.ornl.gov kernel: [1917000.635407] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4h8.ccs.ornl.gov kernel: [1916993.960695] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4g3.ccs.ornl.gov kernel: [1917014.125745] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880ba79d2000 x1471904517610508/t0(0) o38->atlas2-MDT0000-lwp-OST007a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4g3.ccs.ornl.gov kernel: [1917014.218354] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3b3.ccs.ornl.gov kernel: [1916996.000062] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3a6.ccs.ornl.gov kernel: [1917005.614196] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880322c5fc00 x1471904512394920/t0(0) o38->atlas2-MDT0000-lwp-OST0005@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3a6.ccs.ornl.gov kernel: [1917005.703936] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3a5.ccs.ornl.gov kernel: [1916996.532640] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880593040c00 x1471904511738744/t0(0) o38->atlas2-MDT0000-lwp-OST0004@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3i4.ccs.ornl.gov kernel: [1917008.888275] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8802a2393c00 x1471904510470656/t0(0) o38->atlas2-MDT0000-lwp-OST0043@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3i4.ccs.ornl.gov kernel: [1917008.973240] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3b5.ccs.ornl.gov kernel: [1917016.034087] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88023f372000 x1471904514717244/t0(0) o38->atlas2-MDT0000-lwp-OST000c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3b5.ccs.ornl.gov kernel: [1917016.120231] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4f6.ccs.ornl.gov kernel: [1916997.790219] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880a5962e800 x1471904511708836/t0(0) o38->atlas2-MDT0000-lwp-OST0075@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3h4.ccs.ornl.gov kernel: [1917006.815795] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88033ba5a000 x1471904509245540/t0(0) o38->atlas2-MDT0000-lwp-OST003b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4b7.ccs.ornl.gov kernel: [1916997.092670] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880e137cb800 x1471904509377568/t0(0) o38->atlas2-MDT0000-lwp-OST0056@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3h2.ccs.ornl.gov kernel: [1917006.075724] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8808ef710c00 x1471904518760784/t0(0) o38->atlas2-MDT0000-lwp-OST0039@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3h2.ccs.ornl.gov kernel: [1917006.168745] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4b8.ccs.ornl.gov kernel: [1917004.898292] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880e55c34800 x1471904512564188/t0(0) o38->atlas2-MDT0000-lwp-OST0057@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4b8.ccs.ornl.gov kernel: [1917004.988949] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3e7.ccs.ornl.gov kernel: [1917005.443248] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880a6626e800 x1471904508195488/t0(0) o38->atlas2-MDT0000-lwp-OST0026@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3a5.ccs.ornl.gov kernel: [1916996.624920] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4f6.ccs.ornl.gov kernel: [1916997.881013] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3h1.ccs.ornl.gov kernel: [1916991.767495] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88023b599000 x1471904517766176/t0(0) o38->atlas2-MDT0000-lwp-OST0038@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3h1.ccs.ornl.gov kernel: [1916991.852901] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4e8.ccs.ornl.gov kernel: [1917016.034957] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880706b15800 x1471904518473072/t0(0) o38->atlas2-MDT0000-lwp-OST006f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4e8.ccs.ornl.gov kernel: [1917016.122110] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3h4.ccs.ornl.gov kernel: [1917006.904574] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4b7.ccs.ornl.gov kernel: [1916997.185137] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3e7.ccs.ornl.gov kernel: [1917005.530355] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4f1.ccs.ornl.gov kernel: [1917007.256036] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8805b063d400 x1471904513371340/t0(0) o38->atlas2-MDT0000-lwp-OST0070@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4i8.ccs.ornl.gov kernel: [1917016.781620] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88083676c800 x1471904520247404/t0(0) o38->atlas2-MDT0000-lwp-OST008f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4i8.ccs.ornl.gov kernel: [1917016.870921] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3d8.ccs.ornl.gov kernel: [1917009.658343] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880b1fe86000 x1471904508710948/t0(0) o38->atlas2-MDT0000-lwp-OST001f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4b5.ccs.ornl.gov kernel: [1917020.684691] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8802322cb800 x1471904518365936/t0(0) o38->atlas2-MDT0000-lwp-OST0054@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4b5.ccs.ornl.gov kernel: [1917020.770442] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4i6.ccs.ornl.gov kernel: [1917016.506442] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8806ef6a2800 x1471904510597856/t0(0) o38->atlas2-MDT0000-lwp-OST008d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4i6.ccs.ornl.gov kernel: [1917016.594971] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4f1.ccs.ornl.gov kernel: [1917007.344894] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4f4.ccs.ornl.gov kernel: [1917003.582755] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8801e1d41000 x1471904516980536/t0(0) o38->atlas2-MDT0000-lwp-OST0073@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4f4.ccs.ornl.gov kernel: [1917003.668710] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3d8.ccs.ornl.gov kernel: [1917009.744059] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4a6.ccs.ornl.gov kernel: [1917005.205323] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8803b9213400 x1471904512503048/t0(0) o38->atlas2-MDT0000-lwp-OST004d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4a6.ccs.ornl.gov kernel: [1917005.294595] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4d6.ccs.ornl.gov kernel: [1917003.234581] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88058949ec00 x1471904519610692/t0(0) o38->atlas2-MDT0000-lwp-OST0065@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4g1.ccs.ornl.gov kernel: [1917000.227066] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8801baae5400 x1471904508970672/t0(0) o38->atlas2-MDT0000-lwp-OST0078@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3e5.ccs.ornl.gov kernel: [1917004.396351] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880f4bdcb400 x1471904510655080/t0(0) o38->atlas2-MDT0000-lwp-OST0024@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3e5.ccs.ornl.gov kernel: [1917004.486980] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3c7.ccs.ornl.gov kernel: [1917004.038729] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8807e1bc6c00 x1471904510248952/t0(0) o38->atlas2-MDT0000-lwp-OST0016@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3e4.ccs.ornl.gov kernel: [1917017.430205] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8807506a1400 x1471904508538300/t0(0) o38->atlas2-MDT0000-lwp-OST0023@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3e4.ccs.ornl.gov kernel: [1917017.523143] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4a3.ccs.ornl.gov kernel: [1917019.152544] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8803cf3c9000 x1471904514689168/t0(0) o38->atlas2-MDT0000-lwp-OST004a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4a3.ccs.ornl.gov kernel: [1917019.243184] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4d6.ccs.ornl.gov kernel: [1917003.319895] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3d3.ccs.ornl.gov kernel: [1917010.008563] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8802db638c00 x1471904510125996/t0(0) o38->atlas2-MDT0000-lwp-OST001a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3c8.ccs.ornl.gov kernel: [1917008.233024] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880c5550cc00 x1471904507928320/t0(0) o38->atlas2-MDT0000-lwp-OST0017@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3d3.ccs.ornl.gov kernel: [1917010.101746] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4a7.ccs.ornl.gov kernel: [1917014.853678] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880493d1e400 x1471904510936296/t0(0) o38->atlas2-MDT0000-lwp-OST004e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4g1.ccs.ornl.gov kernel: [1917000.315368] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4e6.ccs.ornl.gov kernel: [1917018.958015] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880ef5b07000 x1471904508408396/t0(0) o38->atlas2-MDT0000-lwp-OST006d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4e6.ccs.ornl.gov kernel: [1917019.043106] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4a1.ccs.ornl.gov kernel: [1917001.134738] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880d6a7b8000 x1471904510346080/t0(0) o38->atlas2-MDT0000-lwp-OST0048@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4a1.ccs.ornl.gov kernel: [1917001.227445] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3h6.ccs.ornl.gov kernel: [1917003.576440] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880bc6a70800 x1471904512216800/t0(0) o38->atlas2-MDT0000-lwp-OST003d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3h6.ccs.ornl.gov kernel: [1917003.663961] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3c7.ccs.ornl.gov kernel: [1917004.132134] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3d2.ccs.ornl.gov kernel: [1917004.918927] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880e9bdde000 x1471904511214924/t0(0) o38->atlas2-MDT0000-lwp-OST0019@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3d2.ccs.ornl.gov kernel: [1917005.003801] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3i6.ccs.ornl.gov kernel: [1917002.066383] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8800632fd000 x1471904509634280/t0(0) o38->atlas2-MDT0000-lwp-OST0045@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3i6.ccs.ornl.gov kernel: [1917002.153460] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3c8.ccs.ornl.gov kernel: [1917008.321026] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3c4.ccs.ornl.gov kernel: [1916998.499865] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8801b7d04c00 x1471904511833196/t0(0) o38->atlas2-MDT0000-lwp-OST0013@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3g3.ccs.ornl.gov kernel: [1915600.608407] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880196b01400 x1471904516977036/t0(0) o38->atlas2-MDT0000-lwp-OST0032@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3g3.ccs.ornl.gov kernel: [1915600.699182] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3i3.ccs.ornl.gov kernel: [1916998.696295] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88104e1e2000 x1471904518169548/t0(0) o38->atlas2-MDT0000-lwp-OST0042@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4d1.ccs.ornl.gov kernel: [1916999.247343] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880eddc78400 x1471904516811984/t0(0) o38->atlas2-MDT0000-lwp-OST0060@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4d1.ccs.ornl.gov kernel: [1916999.339704] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4a7.ccs.ornl.gov kernel: [1917014.942606] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3a7.ccs.ornl.gov kernel: [1917001.353242] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88035496fc00 x1471904518655904/t0(0) o38->atlas2-MDT0000-lwp-OST0006@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3a7.ccs.ornl.gov kernel: [1917001.443184] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3g6.ccs.ornl.gov kernel: [1915611.615096] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88090cd43400 x1471904523474104/t0(0) o38->atlas2-MDT0000-lwp-OST0035@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4h3.ccs.ornl.gov kernel: [1917015.934065] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880c33530c00 x1471904509651120/t0(0) o38->atlas2-MDT0000-lwp-OST0082@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3c3.ccs.ornl.gov kernel: [1916995.926654] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88006a108800 x1471904511051720/t0(0) o38->atlas2-MDT0000-lwp-OST0012@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3d7.ccs.ornl.gov kernel: [1917004.111966] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880ddac9cc00 x1471904512149724/t0(0) o38->atlas2-MDT0000-lwp-OST001e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4i1.ccs.ornl.gov kernel: [1917003.360421] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88025f61c000 x1471904513320004/t0(0) o38->atlas2-MDT0000-lwp-OST0088@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4i1.ccs.ornl.gov kernel: [1917003.453697] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3f1.ccs.ornl.gov kernel: [1917007.086754] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8809b7c0f800 x1471904508988704/t0(0) o38->atlas2-MDT0000-lwp-OST0028@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4f8.ccs.ornl.gov kernel: [1917002.500879] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880592785000 x1471904511806304/t0(0) o38->atlas2-MDT0000-lwp-OST0077@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3c4.ccs.ornl.gov kernel: [1916998.593297] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4f8.ccs.ornl.gov kernel: [1917002.589302] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4a8.ccs.ornl.gov kernel: [1917009.294719] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880fd8ce7400 x1471904507751040/t0(0) o38->atlas2-MDT0000-lwp-OST004f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4a8.ccs.ornl.gov kernel: [1917009.387678] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3b7.ccs.ornl.gov kernel: [1917008.144148] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880b42df7800 x1471904519392916/t0(0) o38->atlas2-MDT0000-lwp-OST000e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3b7.ccs.ornl.gov kernel: [1917008.239626] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3i3.ccs.ornl.gov kernel: [1916998.781789] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3g6.ccs.ornl.gov kernel: [1915611.701770] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3i1.ccs.ornl.gov kernel: [1917003.418708] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff8801fc916800 x1471904519755680/t0(0) o38->atlas2-MDT0000-lwp-OST0040@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3i1.ccs.ornl.gov kernel: [1917003.509835] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4h3.ccs.ornl.gov kernel: [1917016.027714] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss4d8.ccs.ornl.gov kernel: [1917016.342234] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff88037c27e000 x1471904516223268/t0(0) o38->atlas2-MDT0000-lwp-OST0067@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss4d8.ccs.ornl.gov kernel: [1917016.431207] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3c3.ccs.ornl.gov kernel: [1916996.020161] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3d7.ccs.ornl.gov kernel: [1917004.204812] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3f1.ccs.ornl.gov kernel: [1917007.176805] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3b8.ccs.ornl.gov kernel: [1917017.468877] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880fb607ac00 x1471904516407052/t0(0) o38->atlas2-MDT0000-lwp-OST000f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3b8.ccs.ornl.gov kernel: [1917017.553184] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3d5.ccs.ornl.gov kernel: [1917006.818585] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880aaa8b3800 x1471904509642304/t0(0) o38->atlas2-MDT0000-lwp-OST001c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3d5.ccs.ornl.gov kernel: [1917006.903619] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:07 atlas-oss3b4.ccs.ornl.gov kernel: [1917014.991410] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632147/real 1405632247] req@ffff880c5d749800 x1471904510548832/t0(0) o38->atlas2-MDT0000-lwp-OST000b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632427 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:07 atlas-oss3b4.ccs.ornl.gov kernel: [1917015.081479] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:53 atlas-oss3i5.ccs.ornl.gov kernel: [703357.361383] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632193/real 1405632293] req@ffff88078dcb7400 x1473175721933608/t0(0) o38->atlas2-MDT0000-lwp-OST00d4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632473 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:53 atlas-oss3i5.ccs.ornl.gov kernel: [703357.448971] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:53 atlas-oss3a8.ccs.ornl.gov kernel: [703358.292200] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632193/real 1405632293] req@ffff8807fb57a000 x1473175720195192/t0(0) o38->atlas2-MDT0000-lwp-OST0097@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632473 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:53 atlas-oss3a8.ccs.ornl.gov kernel: [703358.378002] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:24:54 atlas-oss3d4.ccs.ornl.gov kernel: [703357.845881] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632193/real 1405632293] req@ffff8801c484fc00 x1473175723812988/t0(0) o38->atlas2-MDT0000-lwp-OST013b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632473 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:24:54 atlas-oss3d4.ccs.ornl.gov kernel: [703357.938000] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4e4.ccs.ornl.gov kernel: [1917161.452146] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632296/real 1405632396] req@ffff880192490800 x1471904515374616/t0(0) o38->atlas2-MDT0000-lwp-OST006b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632576 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4e4.ccs.ornl.gov kernel: [1917161.538293] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:26:37 atlas-oss4g5.ccs.ornl.gov kernel: [1917152.854162] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632296/real 1405632396] req@ffff880730429800 x1471904510457032/t0(0) o38->atlas2-MDT0000-lwp-OST007c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632576 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4g5.ccs.ornl.gov kernel: [1917152.944598] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4g2.ccs.ornl.gov kernel: [1917163.537113] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632296/real 1405632396] req@ffff880f36966800 x1471904509229264/t0(0) o38->atlas2-MDT0000-lwp-OST0079@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632576 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4h7.ccs.ornl.gov kernel: [1917161.053753] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632296/real 1405632396] req@ffff88064a87a000 x1471904512615444/t0(0) o38->atlas2-MDT0000-lwp-OST0086@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632576 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4d2.ccs.ornl.gov kernel: [1917157.870786] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff880358f14400 x1471904512221264/t0(0) o38->atlas2-MDT0000-lwp-OST02a1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4i7.ccs.ornl.gov kernel: [1917166.069111] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632296/real 1405632397] req@ffff880354f3a000 x1471904511375400/t0(0) o38->atlas2-MDT0000-lwp-OST008e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632576 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4i7.ccs.ornl.gov kernel: [1917166.154621] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4i2.ccs.ornl.gov kernel: [1917175.664028] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632296/real 1405632397] req@ffff880480057400 x1471904514634824/t0(0) o38->atlas2-MDT0000-lwp-OST0089@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632576 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4i2.ccs.ornl.gov kernel: [1917175.757499] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4b3.ccs.ornl.gov kernel: [1917147.083405] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632296/real 1405632397] req@ffff88073218f400 x1471904509180812/t0(0) o38->atlas2-MDT0000-lwp-OST0052@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632576 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4b3.ccs.ornl.gov kernel: [1917147.174477] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4h6.ccs.ornl.gov kernel: [1917151.572148] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632296/real 1405632397] req@ffff880d59b7b000 x1471904515532392/t0(0) o38->atlas2-MDT0000-lwp-OST0085@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632576 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4f7.ccs.ornl.gov kernel: [1917151.857978] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632296/real 1405632397] req@ffff88058129b400 x1471904510455344/t0(0) o38->atlas2-MDT0000-lwp-OST0076@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632576 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4g2.ccs.ornl.gov kernel: [1917163.630727] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4h2.ccs.ornl.gov kernel: [1917174.037272] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632296/real 1405632397] req@ffff8804ab116c00 x1471904508411428/t0(0) o38->atlas2-MDT0000-lwp-OST0081@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632576 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4d2.ccs.ornl.gov kernel: [1917157.960582] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4h7.ccs.ornl.gov kernel: [1917161.143611] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4h2.ccs.ornl.gov kernel: [1917174.126159] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4h6.ccs.ornl.gov kernel: [1917151.672499] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4f7.ccs.ornl.gov kernel: [1917151.946383] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4c4.ccs.ornl.gov kernel: [1917164.752423] Lustre: 25786:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff88091d498400 x1471904510981348/t0(0) o38->atlas2-MDT0000-lwp-OST005b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3f4.ccs.ornl.gov kernel: [1917155.085217] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff880a68b16400 x1471904507778764/t0(0) o38->atlas2-MDT0000-lwp-OST002b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4c4.ccs.ornl.gov kernel: [1917164.845917] Lustre: 25786:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:26:37 atlas-oss3f4.ccs.ornl.gov kernel: [1917155.172628] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:26:37 atlas-oss4f3.ccs.ornl.gov kernel: [1917147.809236] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff8801aac17000 x1471904513977432/t0(0) o38->atlas2-MDT0000-lwp-OST0072@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4f3.ccs.ornl.gov kernel: [1917147.900724] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss3g1.ccs.ornl.gov kernel: [1915743.062938] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff88062571d400 x1471904510299916/t0(0) o38->atlas2-MDT0000-lwp-OST0030@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3g1.ccs.ornl.gov kernel: [1915743.151569] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss3g2.ccs.ornl.gov kernel: [1915749.745877] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff8807ca3af800 x1471904509036420/t0(0) o38->atlas2-MDT0000-lwp-OST01e1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3g2.ccs.ornl.gov kernel: [1915749.838951] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss3a2.ccs.ornl.gov kernel: [1917152.656013] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff880fecbdbc00 x1471904510524568/t0(0) o38->atlas2-MDT0000-lwp-OST0001@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3a2.ccs.ornl.gov kernel: [1917152.741506] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4d5.ccs.ornl.gov kernel: [1917165.648830] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff8806e72fa000 x1471904519137108/t0(0) o38->atlas2-MDT0000-lwp-OST0064@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4d5.ccs.ornl.gov kernel: [1917165.742290] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4e3.ccs.ornl.gov kernel: [1917158.949607] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff880bb9e2b000 x1471904515524008/t0(0) o38->atlas2-MDT0000-lwp-OST00fa@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3i8.ccs.ornl.gov kernel: [1917167.048469] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff880d3383a800 x1471904515355756/t0(0) o38->atlas2-MDT0000-lwp-OST0047@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3i8.ccs.ornl.gov kernel: [1917167.135655] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4i5.ccs.ornl.gov kernel: [1917161.100240] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff880817510400 x1471904512720692/t0(0) o38->atlas2-MDT0000-lwp-OST008c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4i5.ccs.ornl.gov kernel: [1917161.192312] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4h5.ccs.ornl.gov kernel: [1917155.661761] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff88081f39b400 x1471904509313748/t0(0) o38->atlas2-MDT0000-lwp-OST0084@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3f2.ccs.ornl.gov kernel: [1917148.288189] Lustre: 25710:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff880b05a5e800 x1471904513672816/t0(0) o38->atlas2-MDT0000-lwp-OST0029@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4h5.ccs.ornl.gov kernel: [1917155.754782] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss3f2.ccs.ornl.gov kernel: [1917148.379330] Lustre: 25710:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Jul 17 17:26:37 atlas-oss3a4.ccs.ornl.gov kernel: [1917143.138966] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff881005b54800 x1471904512539068/t0(0) o38->atlas2-MDT0000-lwp-OST0003@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4e3.ccs.ornl.gov kernel: [1917159.042807] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss3d1.ccs.ornl.gov kernel: [1917145.837831] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff88078888cc00 x1471904515778596/t0(0) o38->atlas2-MDT0000-lwp-OST0018@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss4h1.ccs.ornl.gov kernel: [1917147.051815] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff8804053dfc00 x1471904511603420/t0(0) o38->atlas2-MDT0000-lwp-OST0230@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3b6.ccs.ornl.gov kernel: [1917151.407983] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff88061d903400 x1471904512275664/t0(0) o38->atlas2-MDT0000-lwp-OST000d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3b6.ccs.ornl.gov kernel: [1917151.500311] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss3a3.ccs.ornl.gov kernel: [1917150.418989] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff881051749400 x1471904526380200/t0(0) o38->atlas2-MDT0000-lwp-OST0002@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3e8.ccs.ornl.gov kernel: [1917153.298039] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff8806af2a0000 x1471904515139872/t0(0) o38->atlas2-MDT0000-lwp-OST00b7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3e8.ccs.ornl.gov kernel: [1917153.395437] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss3b2.ccs.ornl.gov kernel: [1917153.184184] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff880cb5255800 x1471904511917288/t0(0) o38->atlas2-MDT0000-lwp-OST0009@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3a4.ccs.ornl.gov kernel: [1917143.232752] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss3g4.ccs.ornl.gov kernel: [1915763.713292] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff88011306f800 x1471904510791652/t0(0) o38->atlas2-MDT0000-lwp-OST00c3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3g4.ccs.ornl.gov kernel: [1915763.797813] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss3c2.ccs.ornl.gov kernel: [1917155.256390] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff8805e54a8c00 x1471904511648220/t0(0) o38->atlas2-MDT0000-lwp-OST0011@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3c5.ccs.ornl.gov kernel: [1917159.820536] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff880adee96c00 x1471904512466408/t0(0) o38->atlas2-MDT0000-lwp-OST0014@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3f3.ccs.ornl.gov kernel: [1917166.748141] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff8807cba0a800 x1471904510688384/t0(0) o38->atlas2-MDT0000-lwp-OST01da@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3f3.ccs.ornl.gov kernel: [1917166.840629] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:26:37 atlas-oss3d1.ccs.ornl.gov kernel: [1917145.931305] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4h1.ccs.ornl.gov kernel: [1917147.144725] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss3b2.ccs.ornl.gov kernel: [1917153.277471] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss3c6.ccs.ornl.gov kernel: [1917145.470479] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff880f2397a400 x1471904508187160/t0(0) o38->atlas2-MDT0000-lwp-OST0015@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3a3.ccs.ornl.gov kernel: [1917150.509702] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4c6.ccs.ornl.gov kernel: [1917154.150373] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff880c04556c00 x1471904517937512/t0(0) o38->atlas2-MDT0000-lwp-OST00ed@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3c5.ccs.ornl.gov kernel: [1917159.917248] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss3c2.ccs.ornl.gov kernel: [1917155.345937] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss3c6.ccs.ornl.gov kernel: [1917145.559524] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss4c6.ccs.ornl.gov kernel: [1917154.243112] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:26:37 atlas-oss3i2.ccs.ornl.gov kernel: [1917159.588141] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632297/real 1405632397] req@ffff8802bf385000 x1471904514016544/t0(0) o38->atlas2-MDT0000-lwp-OST0041@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632577 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:26:37 atlas-oss3i2.ccs.ornl.gov kernel: [1917159.673268] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:29:07 atlas-oss4c7.ccs.ornl.gov kernel: [1917303.090438] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632447/real 1405632547] req@ffff8802e966b800 x1471904514425932/t0(0) o38->atlas2-MDT0000-lwp-OST005e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405632727 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:29:07 atlas-oss4c7.ccs.ornl.gov kernel: [1917303.180172] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:34:07 atlas-oss4b2.ccs.ornl.gov kernel: [1917625.804850] Lustre: 27461:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632747/real 1405632847] req@ffff880fedcf7400 x1471904518026208/t0(0) o38->atlas2-MDT0000-lwp-OST0051@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:34:07 atlas-oss4b2.ccs.ornl.gov kernel: [1917625.895273] Lustre: 27461:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:34:07 atlas-oss3h5.ccs.ornl.gov kernel: [1917622.748750] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632747/real 1405632847] req@ffff880815e36400 x1471904507806948/t0(0) o38->atlas2-MDT0000-lwp-OST003c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:34:07 atlas-oss3h5.ccs.ornl.gov kernel: [1917622.838901] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:34:07 atlas-oss4c1.ccs.ornl.gov kernel: [1917608.660475] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632747/real 1405632847] req@ffff88021ba38000 x1471904516097864/t0(0) o38->atlas2-MDT0000-lwp-OST0058@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:34:07 atlas-oss4c1.ccs.ornl.gov kernel: [1917608.749236] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:34:07 atlas-oss3g7.ccs.ornl.gov kernel: [1916200.725053] Lustre: 13449:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632747/real 1405632847] req@ffff880258f5e000 x1471904520422608/t0(0) o38->atlas2-MDT0000-lwp-OST0036@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:34:07 atlas-oss3g7.ccs.ornl.gov kernel: [1916200.818506] Lustre: 13449:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:34:07 atlas-oss4f5.ccs.ornl.gov kernel: [1917601.130315] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632747/real 1405632847] req@ffff880a5188b400 x1471904509391288/t0(0) o38->atlas2-MDT0000-lwp-OST0074@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:34:07 atlas-oss4f5.ccs.ornl.gov kernel: [1917601.219329] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:34:07 atlas-oss4g4.ccs.ornl.gov kernel: [1917617.709318] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632747/real 1405632847] req@ffff8806a8038c00 x1471904515531028/t0(0) o38->atlas2-MDT0000-lwp-OST007b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:34:07 atlas-oss4g4.ccs.ornl.gov kernel: [1917617.796740] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:34:07 atlas-oss4g6.ccs.ornl.gov kernel: [1917612.941208] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632747/real 1405632847] req@ffff8803f7f26400 x1471904507899048/t0(0) o38->atlas2-MDT0000-lwp-OST007d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:34:07 atlas-oss4g6.ccs.ornl.gov kernel: [1917613.034375] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:34:07 atlas-oss4g7.ccs.ornl.gov kernel: [1917625.569705] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632747/real 1405632847] req@ffff880de1bec000 x1471904508873936/t0(0) o38->atlas2-MDT0000-lwp-OST007e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:34:07 atlas-oss4g7.ccs.ornl.gov kernel: [1917625.663241] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:34:07 atlas-oss4e5.ccs.ornl.gov kernel: [1917604.438825] Lustre: 24644:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632747/real 1405632847] req@ffff88034d5a6000 x1471904515279564/t0(0) o38->atlas2-MDT0000-lwp-OST006c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:34:07 atlas-oss4e5.ccs.ornl.gov kernel: [1917604.529715] Lustre: 24644:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:34:07 atlas-oss3e2.ccs.ornl.gov kernel: [1917602.800214] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632747/real 1405632847] req@ffff88037e94cc00 x1471904513179548/t0(0) o38->atlas2-MDT0000-lwp-OST0021@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:34:07 atlas-oss3e2.ccs.ornl.gov kernel: [1917602.885664] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:34:07 atlas-oss4b4.ccs.ornl.gov kernel: [1917601.571734] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632747/real 1405632847] req@ffff88055e58cc00 x1471904528872372/t0(0) o38->atlas2-MDT0000-lwp-OST0053@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:34:07 atlas-oss4b4.ccs.ornl.gov kernel: [1917601.656609] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:34:07 atlas-oss3e6.ccs.ornl.gov kernel: [1917601.032469] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632747/real 1405632847] req@ffff88091b48a800 x1471904520534992/t0(0) o38->atlas2-MDT0000-lwp-OST0025@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:34:07 atlas-oss3b1.ccs.ornl.gov kernel: [1917610.144259] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632747/real 1405632847] req@ffff8802eeaf7000 x1471904516111104/t0(0) o38->atlas2-MDT0000-lwp-OST0008@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633027 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:34:07 atlas-oss3b1.ccs.ornl.gov kernel: [1917610.237856] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:34:07 atlas-oss3e6.ccs.ornl.gov kernel: [1917601.120888] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:36:35 atlas-mds1.ccs.ornl.gov kernel: [797764.732586] LustreError: 7952:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 198210730292, new: 198210730291 replay: 0. see LU-617. Jul 17 17:36:35 atlas-mds1.ccs.ornl.gov kernel: [797764.750412] LustreError: 7952:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Skipped 9 previous similar messages Jul 17 17:36:36 atlas-mds1.ccs.ornl.gov kernel: [797765.824548] LustreError: 15491:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 198210745684, new: 198210745683 replay: 0. see LU-617. Jul 17 17:36:37 atlas-oss3e3.ccs.ornl.gov kernel: [1917749.729158] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8803ecada800 x1471904510468820/t0(0) o38->atlas2-MDT0000-lwp-OST0022@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3e3.ccs.ornl.gov kernel: [1917749.816284] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4i3.ccs.ornl.gov kernel: [1917745.288738] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880d4e943c00 x1471904523159956/t0(0) o38->atlas2-MDT0000-lwp-OST008a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4d7.ccs.ornl.gov kernel: [1917753.774487] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8809ab91a400 x1471904514720356/t0(0) o38->atlas2-MDT0000-lwp-OST00f6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4i4.ccs.ornl.gov kernel: [1917757.990664] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880db2243000 x1471904516513804/t0(0) o38->atlas2-MDT0000-lwp-OST011b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3f6.ccs.ornl.gov kernel: [804457.669949] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8808d18dc800 x1473076856558952/t0(0) o38->atlas2-MDT0000-lwp-OST002d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3f6.ccs.ornl.gov kernel: [804457.759445] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3c1.ccs.ornl.gov kernel: [1917755.219984] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880565cb4000 x1471904511393340/t0(0) o38->atlas2-MDT0000-lwp-OST0010@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3c1.ccs.ornl.gov kernel: [1917755.310022] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3i7.ccs.ornl.gov kernel: [1917750.237136] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff881026d20400 x1471904508389668/t0(0) o38->atlas2-MDT0000-lwp-OST0046@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3i7.ccs.ornl.gov kernel: [1917750.326966] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:36:37 atlas-oss4i3.ccs.ornl.gov kernel: [1917745.375475] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:36:37 atlas-oss4f2.ccs.ornl.gov kernel: [1917756.664042] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff881005c9d800 x1471904508911056/t0(0) o38->atlas2-MDT0000-lwp-OST0101@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4d3.ccs.ornl.gov kernel: [1917749.565363] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880995977800 x1471904516978640/t0(0) o38->atlas2-MDT0000-lwp-OST00f2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4d7.ccs.ornl.gov kernel: [1917753.867516] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4h4.ccs.ornl.gov kernel: [1917772.532435] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8804d0812400 x1471904512334728/t0(0) o38->atlas2-MDT0000-lwp-OST0083@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4c3.ccs.ornl.gov kernel: [1917749.888016] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880aed21e400 x1471904512246448/t0(0) o38->atlas2-MDT0000-lwp-OST00ea@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4c5.ccs.ornl.gov kernel: [1917748.847246] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8805af35bc00 x1471904510171592/t0(0) o38->atlas2-MDT0000-lwp-OST005c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4i4.ccs.ornl.gov kernel: [1917758.078012] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3d6.ccs.ornl.gov kernel: [1917767.524566] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880cc672f000 x1471904511183748/t0(0) o38->atlas2-MDT0000-lwp-OST001d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4a4.ccs.ornl.gov kernel: [1917754.676550] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88004e072800 x1471904509591476/t0(0) o38->atlas2-MDT0000-lwp-OST01fb@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4a4.ccs.ornl.gov kernel: [1917754.761510] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3g8.ccs.ornl.gov kernel: [1916351.169943] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880804ad7800 x1471904513602796/t0(0) o38->atlas2-MDT0000-lwp-OST01e7@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3g8.ccs.ornl.gov kernel: [1916351.261200] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3h8.ccs.ornl.gov kernel: [1917749.985195] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880818c35800 x1471904514773188/t0(0) o38->atlas2-MDT0000-lwp-OST003f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4a5.ccs.ornl.gov kernel: [1917772.118644] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88074f72f000 x1471904519454512/t0(0) o38->atlas2-MDT0000-lwp-OST004c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4f2.ccs.ornl.gov kernel: [1917756.753798] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3e1.ccs.ornl.gov kernel: [1917755.893633] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8809b4d88400 x1471904509694716/t0(0) o38->atlas2-MDT0000-lwp-OST00b0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4c5.ccs.ornl.gov kernel: [1917748.940670] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4d3.ccs.ornl.gov kernel: [1917749.656677] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:36:37 atlas-oss4c8.ccs.ornl.gov kernel: [1917745.002715] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88029656f800 x1471904512197284/t0(0) o38->atlas2-MDT0000-lwp-OST005f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4a2.ccs.ornl.gov kernel: [1917767.790206] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8806efc4e800 x1471904510896076/t0(0) o38->atlas2-MDT0000-lwp-OST0049@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4a2.ccs.ornl.gov kernel: [1917767.874373] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4e7.ccs.ornl.gov kernel: [1917750.548918] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880165ec1c00 x1471904522621896/t0(0) o38->atlas2-MDT0000-lwp-OST021e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4e7.ccs.ornl.gov kernel: [1917750.634202] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:36:37 atlas-oss3f5.ccs.ornl.gov kernel: [804445.037367] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880bb5a0c800 x1473076856406664/t0(0) o38->atlas2-MDT0000-lwp-OST002c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3f5.ccs.ornl.gov kernel: [804445.123486] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:36:37 atlas-oss4h4.ccs.ornl.gov kernel: [1917772.622194] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4c3.ccs.ornl.gov kernel: [1917749.975003] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3h8.ccs.ornl.gov kernel: [1917750.078306] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3d6.ccs.ornl.gov kernel: [1917767.609338] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3f8.ccs.ornl.gov kernel: [804455.548170] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880fea770c00 x1473076861321312/t0(0) o38->atlas2-MDT0000-lwp-OST00bf@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3f8.ccs.ornl.gov kernel: [804455.641478] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3g5.ccs.ornl.gov kernel: [1916354.753864] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8802e0b50c00 x1471904518249452/t0(0) o38->atlas2-MDT0000-lwp-OST0034@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3g5.ccs.ornl.gov kernel: [1916354.841565] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4a5.ccs.ornl.gov kernel: [1917772.203551] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4c2.ccs.ornl.gov kernel: [1917757.096573] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8808ed75f000 x1471904508571032/t0(0) o38->atlas2-MDT0000-lwp-OST0059@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3e1.ccs.ornl.gov kernel: [1917755.978768] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4c8.ccs.ornl.gov kernel: [1917745.088106] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4e1.ccs.ornl.gov kernel: [1917751.042246] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880e4596c000 x1471904515145120/t0(0) o38->atlas2-MDT0000-lwp-OST0068@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3h3.ccs.ornl.gov kernel: [1917748.789741] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8803c00a4400 x1471904512413972/t0(0) o38->atlas2-MDT0000-lwp-OST003a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3a1.ccs.ornl.gov kernel: [1917748.018255] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8810184ea800 x1471904525932132/t0(0) o38->atlas2-MDT0000-lwp-OST0090@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3a1.ccs.ornl.gov kernel: [1917748.106070] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4d4.ccs.ornl.gov kernel: [1917773.560500] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8801af641000 x1471904513679524/t0(0) o38->atlas2-MDT0000-lwp-OST00f3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4d4.ccs.ornl.gov kernel: [1917773.649335] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3h7.ccs.ornl.gov kernel: [1917759.924494] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880520165400 x1471904510333308/t0(0) o38->atlas2-MDT0000-lwp-OST003e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3h7.ccs.ornl.gov kernel: [1917760.012827] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4e2.ccs.ornl.gov kernel: [1917767.024738] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880a8b8f4000 x1471904514146724/t0(0) o38->atlas2-MDT0000-lwp-OST0219@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4e2.ccs.ornl.gov kernel: [1917767.111371] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:36:37 atlas-oss4g8.ccs.ornl.gov kernel: [1917752.287933] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8802a135dc00 x1471904521747196/t0(0) o38->atlas2-MDT0000-lwp-OST010f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4g8.ccs.ornl.gov kernel: [1917752.381285] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4h8.ccs.ornl.gov kernel: [1917744.142956] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8802a3197800 x1471904514267568/t0(0) o38->atlas2-MDT0000-lwp-OST0087@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3f7.ccs.ornl.gov kernel: [804084.274293] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880860664000 x1473076863306556/t0(0) o38->atlas2-MDT0000-lwp-OST002e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4c2.ccs.ornl.gov kernel: [1917757.190344] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3f7.ccs.ornl.gov kernel: [804084.367364] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4b6.ccs.ornl.gov kernel: [1917750.824425] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88079b7c4800 x1471904511213932/t0(0) o38->atlas2-MDT0000-lwp-OST0055@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4b6.ccs.ornl.gov kernel: [1917750.915083] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4e1.ccs.ornl.gov kernel: [1917751.128559] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3b3.ccs.ornl.gov kernel: [1917746.185996] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880710524c00 x1471904513820792/t0(0) o38->atlas2-MDT0000-lwp-OST000a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4b1.ccs.ornl.gov kernel: [1917757.721466] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8805f2407800 x1471904507949456/t0(0) o38->atlas2-MDT0000-lwp-OST0050@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3h3.ccs.ornl.gov kernel: [1917748.875643] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4h8.ccs.ornl.gov kernel: [1917744.230635] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4g3.ccs.ornl.gov kernel: [1917764.406361] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880869a03c00 x1471904517610596/t0(0) o38->atlas2-MDT0000-lwp-OST007a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3b3.ccs.ornl.gov kernel: [1917746.277232] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4b1.ccs.ornl.gov kernel: [1917757.815193] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3a5.ccs.ornl.gov kernel: [1917746.783913] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880635f96c00 x1471904511738840/t0(0) o38->atlas2-MDT0000-lwp-OST0124@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3a5.ccs.ornl.gov kernel: [1917746.876185] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:36:37 atlas-oss3a6.ccs.ornl.gov kernel: [1917755.892393] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8801d2a7fc00 x1471904512395008/t0(0) o38->atlas2-MDT0000-lwp-OST0005@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3a6.ccs.ornl.gov kernel: [1917755.984959] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4g3.ccs.ornl.gov kernel: [1917764.492628] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3b5.ccs.ornl.gov kernel: [1917766.316157] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88034bd8d400 x1471904514717332/t0(0) o38->atlas2-MDT0000-lwp-OST000c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3b5.ccs.ornl.gov kernel: [1917766.403537] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3i4.ccs.ornl.gov kernel: [1917759.167396] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88053f6a6000 x1471904510470744/t0(0) o38->atlas2-MDT0000-lwp-OST0043@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3i4.ccs.ornl.gov kernel: [1917759.255945] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4f6.ccs.ornl.gov kernel: [1917748.066910] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880a2f029c00 x1471904511708924/t0(0) o38->atlas2-MDT0000-lwp-OST0075@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4b7.ccs.ornl.gov kernel: [1917747.369161] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880b55ed7c00 x1471904509377656/t0(0) o38->atlas2-MDT0000-lwp-OST0056@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3h2.ccs.ornl.gov kernel: [1917756.354143] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880fecece800 x1471904518760872/t0(0) o38->atlas2-MDT0000-lwp-OST0039@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3h4.ccs.ornl.gov kernel: [1917757.094449] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8805823afc00 x1471904509245632/t0(0) o38->atlas2-MDT0000-lwp-OST00cb@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3h4.ccs.ornl.gov kernel: [1917757.187706] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4b8.ccs.ornl.gov kernel: [1917755.176394] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880d7732b000 x1471904512564276/t0(0) o38->atlas2-MDT0000-lwp-OST0057@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4b8.ccs.ornl.gov kernel: [1917755.269783] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3e7.ccs.ornl.gov kernel: [1917755.720991] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880e9b03fc00 x1471904508195576/t0(0) o38->atlas2-MDT0000-lwp-OST0026@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3e7.ccs.ornl.gov kernel: [1917755.812178] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4f6.ccs.ornl.gov kernel: [1917748.160614] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3h1.ccs.ornl.gov kernel: [1917742.040309] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880772ae8400 x1471904517766264/t0(0) o38->atlas2-MDT0000-lwp-OST0038@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4b7.ccs.ornl.gov kernel: [1917747.454708] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3h2.ccs.ornl.gov kernel: [1917756.442388] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4e8.ccs.ornl.gov kernel: [1917766.315909] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880840d8e800 x1471904518473168/t0(0) o38->atlas2-MDT0000-lwp-OST00ff@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4i8.ccs.ornl.gov kernel: [1917767.062708] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88033074ec00 x1471904520247504/t0(0) o38->atlas2-MDT0000-lwp-OST023f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4i8.ccs.ornl.gov kernel: [1917767.148262] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4f1.ccs.ornl.gov kernel: [1917757.534753] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8803ec831400 x1471904513371428/t0(0) o38->atlas2-MDT0000-lwp-OST0070@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3d8.ccs.ornl.gov kernel: [1917759.939331] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880d78e49c00 x1471904508711052/t0(0) o38->atlas2-MDT0000-lwp-OST01cf@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3h1.ccs.ornl.gov kernel: [1917742.130093] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4b5.ccs.ornl.gov kernel: [1917770.969127] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8805ef27dc00 x1471904518366024/t0(0) o38->atlas2-MDT0000-lwp-OST0054@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4b5.ccs.ornl.gov kernel: [1917771.056206] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4e8.ccs.ornl.gov kernel: [1917766.405315] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:36:37 atlas-oss4f1.ccs.ornl.gov kernel: [1917757.625838] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4i6.ccs.ornl.gov kernel: [1917766.788970] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88017afc7c00 x1471904510597960/t0(0) o38->atlas2-MDT0000-lwp-OST02cd@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4i6.ccs.ornl.gov kernel: [1917766.882203] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4f4.ccs.ornl.gov kernel: [1917753.860047] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8807966eac00 x1471904516980636/t0(0) o38->atlas2-MDT0000-lwp-OST0223@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4f4.ccs.ornl.gov kernel: [1917753.947361] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3f2.ccs.ornl.gov kernel: [1917748.536464] Lustre: 25710:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88094abc7000 x1471904513672900/t0(0) o38->atlas2-MDT0000-lwp-OST00b9@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3f2.ccs.ornl.gov kernel: [1917748.621901] Lustre: 25710:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:36:37 atlas-oss3d8.ccs.ornl.gov kernel: [1917760.029144] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:36:37 atlas-oss3e5.ccs.ornl.gov kernel: [1917754.660113] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880cc9451000 x1471904510655168/t0(0) o38->atlas2-MDT0000-lwp-OST0024@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3e5.ccs.ornl.gov kernel: [1917754.747658] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4a6.ccs.ornl.gov kernel: [1917755.483716] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8803b9213400 x1471904512503136/t0(0) o38->atlas2-MDT0000-lwp-OST004d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4a6.ccs.ornl.gov kernel: [1917755.575765] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4d6.ccs.ornl.gov kernel: [1917753.512270] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880745d9d800 x1471904519610780/t0(0) o38->atlas2-MDT0000-lwp-OST0065@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4d6.ccs.ornl.gov kernel: [1917753.602300] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4g1.ccs.ornl.gov kernel: [1917750.503780] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8806f29dc800 x1471904508970760/t0(0) o38->atlas2-MDT0000-lwp-OST0078@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3f3.ccs.ornl.gov kernel: [1917766.974019] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88037ae76c00 x1471904510688452/t0(0) o38->atlas2-MDT0000-lwp-OST002a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3c7.ccs.ornl.gov kernel: [1917754.316386] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8803d72cd000 x1471904510249044/t0(0) o38->atlas2-MDT0000-lwp-OST0016@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3c7.ccs.ornl.gov kernel: [1917754.403381] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:36:37 atlas-oss3e4.ccs.ornl.gov kernel: [1917767.713514] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88048979f400 x1471904508538388/t0(0) o38->atlas2-MDT0000-lwp-OST0023@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4a3.ccs.ornl.gov kernel: [1917769.436008] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8801f8c0d400 x1471904514689256/t0(0) o38->atlas2-MDT0000-lwp-OST004a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3d3.ccs.ornl.gov kernel: [1917760.288786] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8801a00c6400 x1471904510126088/t0(0) o38->atlas2-MDT0000-lwp-OST00aa@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4g1.ccs.ornl.gov kernel: [1917750.593731] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3c8.ccs.ornl.gov kernel: [1917758.511275] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880a37f4ec00 x1471904507928408/t0(0) o38->atlas2-MDT0000-lwp-OST0017@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3f3.ccs.ornl.gov kernel: [1917767.059182] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Jul 17 17:36:37 atlas-oss4a7.ccs.ornl.gov kernel: [1917765.135477] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88043aa20800 x1471904510936384/t0(0) o38->atlas2-MDT0000-lwp-OST004e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4a7.ccs.ornl.gov kernel: [1917765.228900] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4e6.ccs.ornl.gov kernel: [1917769.241495] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff881003b2a800 x1471904508408484/t0(0) o38->atlas2-MDT0000-lwp-OST006d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4e6.ccs.ornl.gov kernel: [1917769.331256] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4a1.ccs.ornl.gov kernel: [1917751.410914] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880b1db46400 x1471904510346168/t0(0) o38->atlas2-MDT0000-lwp-OST0048@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3h6.ccs.ornl.gov kernel: [1917753.854160] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880dfea35400 x1471904512216888/t0(0) o38->atlas2-MDT0000-lwp-OST003d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3d2.ccs.ornl.gov kernel: [1917755.195438] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880ceea25000 x1471904511215016/t0(0) o38->atlas2-MDT0000-lwp-OST0019@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3e4.ccs.ornl.gov kernel: [1917767.800459] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3c4.ccs.ornl.gov kernel: [1917748.775626] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880661f35c00 x1471904511833284/t0(0) o38->atlas2-MDT0000-lwp-OST0013@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3i6.ccs.ornl.gov kernel: [1917752.342215] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8806bdd44400 x1471904509634368/t0(0) o38->atlas2-MDT0000-lwp-OST0045@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3i6.ccs.ornl.gov kernel: [1917752.434520] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4a3.ccs.ornl.gov kernel: [1917769.522585] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3d3.ccs.ornl.gov kernel: [1917760.377225] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4d1.ccs.ornl.gov kernel: [1917749.523551] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88106f950400 x1471904516812076/t0(0) o38->atlas2-MDT0000-lwp-OST00f0@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4d1.ccs.ornl.gov kernel: [1917749.608860] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3c8.ccs.ornl.gov kernel: [1917758.604550] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3a7.ccs.ornl.gov kernel: [1917751.629559] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88036e6df000 x1471904518655992/t0(0) o38->atlas2-MDT0000-lwp-OST0006@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3g3.ccs.ornl.gov kernel: [1916350.887230] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880598942000 x1471904516977128/t0(0) o38->atlas2-MDT0000-lwp-OST0032@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3i3.ccs.ornl.gov kernel: [1917748.972547] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880bf4dfd000 x1471904518169636/t0(0) o38->atlas2-MDT0000-lwp-OST0042@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4a1.ccs.ornl.gov kernel: [1917751.510051] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3g6.ccs.ornl.gov kernel: [1916361.897595] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8808fb7e1000 x1471904523474192/t0(0) o38->atlas2-MDT0000-lwp-OST0035@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4h3.ccs.ornl.gov kernel: [1917766.215275] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88090f8f2800 x1471904509651208/t0(0) o38->atlas2-MDT0000-lwp-OST0082@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3d2.ccs.ornl.gov kernel: [1917755.284531] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:36:37 atlas-oss3c3.ccs.ornl.gov kernel: [1917746.202043] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8806af061c00 x1471904511051824/t0(0) o38->atlas2-MDT0000-lwp-OST01c2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3c3.ccs.ornl.gov kernel: [1917746.288901] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:36:37 atlas-oss3d7.ccs.ornl.gov kernel: [1917754.389499] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880b5d796c00 x1471904512149812/t0(0) o38->atlas2-MDT0000-lwp-OST001e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3d7.ccs.ornl.gov kernel: [1917754.475494] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3h6.ccs.ornl.gov kernel: [1917753.947289] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4i1.ccs.ornl.gov kernel: [1917753.636956] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880659de6c00 x1471904513320108/t0(0) o38->atlas2-MDT0000-lwp-OST0238@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4i1.ccs.ornl.gov kernel: [1917753.725194] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4a8.ccs.ornl.gov kernel: [1917759.574100] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880a1d41e800 x1471904507751128/t0(0) o38->atlas2-MDT0000-lwp-OST004f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3f1.ccs.ornl.gov kernel: [1917757.365593] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8809fd3c2800 x1471904508988792/t0(0) o38->atlas2-MDT0000-lwp-OST0028@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4f8.ccs.ornl.gov kernel: [1917752.777955] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff88025873d800 x1471904511806404/t0(0) o38->atlas2-MDT0000-lwp-OST0227@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4f8.ccs.ornl.gov kernel: [1917752.868475] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3c4.ccs.ornl.gov kernel: [1917748.863090] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3a7.ccs.ornl.gov kernel: [1917751.715363] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3b7.ccs.ornl.gov kernel: [1917758.422715] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880fde74f400 x1471904519393020/t0(0) o38->atlas2-MDT0000-lwp-OST01be@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3b7.ccs.ornl.gov kernel: [1917758.511070] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:36:37 atlas-oss3g3.ccs.ornl.gov kernel: [1916350.980620] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:36:37 atlas-oss3i3.ccs.ornl.gov kernel: [1917749.062876] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4a8.ccs.ornl.gov kernel: [1917759.660357] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3g6.ccs.ornl.gov kernel: [1916361.989191] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3i1.ccs.ornl.gov kernel: [1917753.696804] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff8801e357b800 x1471904519755772/t0(0) o38->atlas2-MDT0000-lwp-OST0040@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4h3.ccs.ornl.gov kernel: [1917766.302709] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss4d8.ccs.ornl.gov kernel: [1917766.624760] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880127db6c00 x1471904516223356/t0(0) o38->atlas2-MDT0000-lwp-OST0067@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss4d8.ccs.ornl.gov kernel: [1917766.715499] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3f1.ccs.ornl.gov kernel: [1917757.455209] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3i1.ccs.ornl.gov kernel: [1917753.790498] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:36:37 atlas-oss3d5.ccs.ornl.gov kernel: [1917757.097374] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880c8a384c00 x1471904509642392/t0(0) o38->atlas2-MDT0000-lwp-OST001c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3d5.ccs.ornl.gov kernel: [1917757.186925] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:36:37 atlas-oss3b8.ccs.ornl.gov kernel: [1917767.777526] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880e287c9c00 x1471904516407148/t0(0) o38->atlas2-MDT0000-lwp-OST000f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3b8.ccs.ornl.gov kernel: [1917767.869796] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Jul 17 17:36:37 atlas-oss3b4.ccs.ornl.gov kernel: [1917765.272906] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632897/real 1405632997] req@ffff880d4508e800 x1471904510548920/t0(0) o38->atlas2-MDT0000-lwp-OST000b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633177 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:36:37 atlas-oss3b4.ccs.ornl.gov kernel: [1917765.375393] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:37:23 atlas-oss3i5.ccs.ornl.gov kernel: [704107.640522] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632943/real 1405633043] req@ffff880871281000 x1473175721933700/t0(0) o38->atlas2-MDT0000-lwp-OST0044@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633223 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:37:23 atlas-oss3i5.ccs.ornl.gov kernel: [704107.732423] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:37:23 atlas-oss3a8.ccs.ornl.gov kernel: [704108.569931] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632943/real 1405633043] req@ffff8807816e2800 x1473175720195284/t0(0) o38->atlas2-MDT0000-lwp-OST0127@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633223 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:37:23 atlas-oss3a8.ccs.ornl.gov kernel: [704108.659050] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:37:24 atlas-oss3d4.ccs.ornl.gov kernel: [704108.124996] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405632943/real 1405633043] req@ffff880fd1f9d000 x1473175723813076/t0(0) o38->atlas2-MDT0000-lwp-OST013b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633223 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:37:24 atlas-oss3d4.ccs.ornl.gov kernel: [704108.210543] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:39:07 atlas-oss4g5.ccs.ornl.gov kernel: [1917903.130133] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633046/real 1405633146] req@ffff88040eca2c00 x1471904510457144/t0(0) o38->atlas2-MDT0000-lwp-OST007c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633326 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4g5.ccs.ornl.gov kernel: [1917903.223229] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss4e4.ccs.ornl.gov kernel: [1917911.732899] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633046/real 1405633146] req@ffff8806165d2000 x1471904515374728/t0(0) o38->atlas2-MDT0000-lwp-OST006b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633326 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4e4.ccs.ornl.gov kernel: [1917911.823460] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss4g2.ccs.ornl.gov kernel: [1917913.818561] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633046/real 1405633146] req@ffff8808f3042800 x1471904509229376/t0(0) o38->atlas2-MDT0000-lwp-OST0079@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633326 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4h7.ccs.ornl.gov kernel: [1917911.332917] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633046/real 1405633146] req@ffff880608192400 x1471904512615552/t0(0) o38->atlas2-MDT0000-lwp-OST0086@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633326 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4h7.ccs.ornl.gov kernel: [1917911.424971] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:39:07 atlas-oss4h6.ccs.ornl.gov kernel: [1917901.847953] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633046/real 1405633147] req@ffff8806d2c6c000 x1471904515532504/t0(0) o38->atlas2-MDT0000-lwp-OST0085@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633326 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4h6.ccs.ornl.gov kernel: [1917901.932820] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss4d2.ccs.ornl.gov kernel: [1917908.149212] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633046/real 1405633147] req@ffff880ed9068c00 x1471904512221364/t0(0) o38->atlas2-MDT0000-lwp-OST0061@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633326 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4i7.ccs.ornl.gov kernel: [1917916.351955] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633046/real 1405633147] req@ffff880871690c00 x1471904511375512/t0(0) o38->atlas2-MDT0000-lwp-OST008e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633326 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4i7.ccs.ornl.gov kernel: [1917916.441769] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss4i2.ccs.ornl.gov kernel: [1917925.950398] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633046/real 1405633147] req@ffff880771d2f400 x1471904514634936/t0(0) o38->atlas2-MDT0000-lwp-OST0089@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633326 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4i2.ccs.ornl.gov kernel: [1917926.037592] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss4b3.ccs.ornl.gov kernel: [1917897.358383] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633046/real 1405633147] req@ffff8802dcffdc00 x1471904509180924/t0(0) o38->atlas2-MDT0000-lwp-OST0052@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633326 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4f7.ccs.ornl.gov kernel: [1917902.135096] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633046/real 1405633147] req@ffff8802b9b4c400 x1471904510455448/t0(0) o38->atlas2-MDT0000-lwp-OST0076@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633326 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4f7.ccs.ornl.gov kernel: [1917902.227906] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Jul 17 17:39:07 atlas-oss4g2.ccs.ornl.gov kernel: [1917913.905654] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss4h2.ccs.ornl.gov kernel: [1917924.322659] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633046/real 1405633147] req@ffff8805fdfafc00 x1471904508411540/t0(0) o38->atlas2-MDT0000-lwp-OST0081@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633326 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4b3.ccs.ornl.gov kernel: [1917897.445149] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss4d2.ccs.ornl.gov kernel: [1917908.241379] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss4h2.ccs.ornl.gov kernel: [1917924.414032] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss4c4.ccs.ornl.gov kernel: [1917915.034361] Lustre: 25786:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff8803308fe000 x1471904510981460/t0(0) o38->atlas2-MDT0000-lwp-OST005b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4c4.ccs.ornl.gov kernel: [1917915.121051] Lustre: 25786:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss3f4.ccs.ornl.gov kernel: [1917905.348381] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff88011c428c00 x1471904507778876/t0(0) o38->atlas2-MDT0000-lwp-OST002b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss3f4.ccs.ornl.gov kernel: [1917905.441865] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss4f3.ccs.ornl.gov kernel: [1917898.083331] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff88064fecb800 x1471904513977544/t0(0) o38->atlas2-MDT0000-lwp-OST0072@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4f3.ccs.ornl.gov kernel: [1917898.176803] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss3g2.ccs.ornl.gov kernel: [1916500.023728] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff88046c9f1c00 x1471904509036520/t0(0) o38->atlas2-MDT0000-lwp-OST0031@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss3g2.ccs.ornl.gov kernel: [1916500.111721] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss3g1.ccs.ornl.gov kernel: [1916493.337890] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff880379213c00 x1471904510300028/t0(0) o38->atlas2-MDT0000-lwp-OST0030@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss3g1.ccs.ornl.gov kernel: [1916493.431186] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss3a2.ccs.ornl.gov kernel: [1917902.921126] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff881016921c00 x1471904510524676/t0(0) o38->atlas2-MDT0000-lwp-OST0001@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4e3.ccs.ornl.gov kernel: [1917909.228212] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff880a447be400 x1471904515524116/t0(0) o38->atlas2-MDT0000-lwp-OST006a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4e3.ccs.ornl.gov kernel: [1917909.314077] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss4d5.ccs.ornl.gov kernel: [1917915.929911] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff8805d0f75400 x1471904519137220/t0(0) o38->atlas2-MDT0000-lwp-OST0064@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4i5.ccs.ornl.gov kernel: [1917911.380253] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff8806ff7d6000 x1471904512720804/t0(0) o38->atlas2-MDT0000-lwp-OST008c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss3a2.ccs.ornl.gov kernel: [1917903.013125] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:39:07 atlas-oss3i8.ccs.ornl.gov kernel: [1917917.331326] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff880a5b70cc00 x1471904515355868/t0(0) o38->atlas2-MDT0000-lwp-OST0047@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss3i8.ccs.ornl.gov kernel: [1917917.423208] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss4h5.ccs.ornl.gov kernel: [1917905.941461] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff8803de976400 x1471904509313860/t0(0) o38->atlas2-MDT0000-lwp-OST0084@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4h5.ccs.ornl.gov kernel: [1917906.028937] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss3a4.ccs.ornl.gov kernel: [1917893.412778] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff880ff4939c00 x1471904512539180/t0(0) o38->atlas2-MDT0000-lwp-OST0003@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4d5.ccs.ornl.gov kernel: [1917916.019626] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss4h1.ccs.ornl.gov kernel: [1917897.326149] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff8806af4aa000 x1471904511603520/t0(0) o38->atlas2-MDT0000-lwp-OST0080@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss3d1.ccs.ornl.gov kernel: [1917896.112716] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff88063c975000 x1471904515778708/t0(0) o38->atlas2-MDT0000-lwp-OST0018@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss3d1.ccs.ornl.gov kernel: [1917896.201932] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss3b6.ccs.ornl.gov kernel: [1917901.684415] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff88056f94e800 x1471904512275776/t0(0) o38->atlas2-MDT0000-lwp-OST000d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4i5.ccs.ornl.gov kernel: [1917911.465247] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss3e8.ccs.ornl.gov kernel: [1917903.574302] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff8804b7e3fc00 x1471904515139980/t0(0) o38->atlas2-MDT0000-lwp-OST0027@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss3a3.ccs.ornl.gov kernel: [1917900.694598] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff880aa7e76400 x1471904526380312/t0(0) o38->atlas2-MDT0000-lwp-OST0002@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss3e8.ccs.ornl.gov kernel: [1917903.666563] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss3b2.ccs.ornl.gov kernel: [1917903.461613] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff88093dc67400 x1471904511917400/t0(0) o38->atlas2-MDT0000-lwp-OST0009@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss3b2.ccs.ornl.gov kernel: [1917903.550999] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss3g4.ccs.ornl.gov kernel: [1916513.997002] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff8804f9639000 x1471904510791756/t0(0) o38->atlas2-MDT0000-lwp-OST0033@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss3g4.ccs.ornl.gov kernel: [1916514.085109] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:39:07 atlas-oss3a4.ccs.ornl.gov kernel: [1917893.503793] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss4h1.ccs.ornl.gov kernel: [1917897.412396] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss3c2.ccs.ornl.gov kernel: [1917905.534390] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff8806bea27000 x1471904511648332/t0(0) o38->atlas2-MDT0000-lwp-OST0011@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss3b6.ccs.ornl.gov kernel: [1917901.769986] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss3c5.ccs.ornl.gov kernel: [1917910.100814] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff880d548fe400 x1471904512466520/t0(0) o38->atlas2-MDT0000-lwp-OST0014@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss3c5.ccs.ornl.gov kernel: [1917910.191798] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss3a3.ccs.ornl.gov kernel: [1917900.780690] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss3c6.ccs.ornl.gov kernel: [1917895.745290] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff88060570dc00 x1471904508187272/t0(0) o38->atlas2-MDT0000-lwp-OST0015@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss4c6.ccs.ornl.gov kernel: [1917904.427871] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff8808c3439c00 x1471904517937620/t0(0) o38->atlas2-MDT0000-lwp-OST005d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss3c2.ccs.ornl.gov kernel: [1917905.625580] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss3c6.ccs.ornl.gov kernel: [1917895.836880] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss4c6.ccs.ornl.gov kernel: [1917904.517088] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:39:07 atlas-oss3i2.ccs.ornl.gov kernel: [1917909.867996] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633047/real 1405633147] req@ffff8802c3137800 x1471904514016656/t0(0) o38->atlas2-MDT0000-lwp-OST0041@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633327 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:39:07 atlas-oss3i2.ccs.ornl.gov kernel: [1917909.957295] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:41:37 atlas-oss4c7.ccs.ornl.gov kernel: [1918053.367730] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633197/real 1405633297] req@ffff8805d79d3800 x1471904514426024/t0(0) o38->atlas2-MDT0000-lwp-OST00ee@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633477 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:41:37 atlas-oss4c7.ccs.ornl.gov kernel: [1918053.461402] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: imklog 5.8.10, log source = /proc/kmsg started. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Initializing cgroup subsys cpuset Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Initializing cgroup subsys cpu Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Linux version 2.6.32-358.23.2.el6.atlas.x86_64 (jsimmons@lgmgmt1) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC) ) #1 SMP Fri Mar 28 07:01:43 EDT 2014 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Command line: initrd=initrd-2.6.32-358.23.2.el6.atlas.x86_64-gedi selinux=0 audit=0 panic=10 console=tty0 console=ttyS1,115200n8 console=ttyS2,115200n8 crashkernel=192M BOOT_IMAGE=vmlinuz-2.6.32-358.23.2.el6.atlas.x86_64 BOOTIF=01-90-b1-1c-38-49-d3 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] KERNEL supported cpus: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Intel GenuineIntel Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] AMD AuthenticAMD Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Centaur CentaurHauls Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] BIOS-provided physical RAM map: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009d000 (usable) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] BIOS-e820: 0000000000100000 - 000000007d2f0000 (usable) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] BIOS-e820: 000000007d2f0000 - 000000007d31c000 (reserved) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] BIOS-e820: 000000007d31c000 - 000000007d35b000 (ACPI data) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] BIOS-e820: 000000007d35b000 - 0000000080000000 (reserved) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] BIOS-e820: 00000000fe000000 - 0000000100000000 (reserved) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] BIOS-e820: 0000000100000000 - 0000004080000000 (usable) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] DMI 2.7 present. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SMBIOS version 2.7 @ 0xFCFB0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] DMI: Dell Inc. PowerEdge R620/0KCKR5, BIOS 2.2.2 01/16/2014 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] last_pfn = 0x4080000 max_arch_pfn = 0x400000000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] MTRR default type: uncachable Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] MTRR fixed ranges enabled: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 00000-9FFFF write-back Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] A0000-BFFFF uncachable Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] C0000-CFFFF write-protect Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] D0000-EBFFF uncachable Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] EC000-FFFFF write-protect Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] MTRR variable ranges enabled: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 0 base 000000000000 mask 3FFF80000000 write-back Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 1 base 000100000000 mask 3FFF00000000 write-back Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 2 base 000200000000 mask 3FFE00000000 write-back Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 3 base 000400000000 mask 3FFC00000000 write-back Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 4 base 000800000000 mask 3FF800000000 write-back Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 5 base 001000000000 mask 3FF000000000 write-back Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 6 base 002000000000 mask 3FE000000000 write-back Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 7 base 004000000000 mask 3FFF80000000 write-back Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 8 disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 9 disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] e820 update range: 0000000080000000 - 0000000100000000 (usable) ==> (reserved) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] last_pfn = 0x7d2f0 max_arch_pfn = 0x400000000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] initial memory mapped : 0 - 20000000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Using GB pages for direct mapping Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] init_memory_mapping: 0000000000000000-000000007d2f0000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 0000000000 - 0040000000 page 1G Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 0040000000 - 007d200000 page 2M Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 007d200000 - 007d2f0000 page 4k Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] kernel direct mapping tables up to 7d2f0000 @ 8000-b000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] init_memory_mapping: 0000000100000000-0000004080000000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 0100000000 - 4080000000 page 1G Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] kernel direct mapping tables up to 4080000000 @ a000-b000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] RAMDISK: 7c594000 - 7d2cf2df Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: RSDP 00000000000f0c70 00024 (v02 DELL ) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: XSDT 00000000000f0db8 000A4 (v01 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: FACP 000000007d33534c 000F4 (v03 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: DSDT 000000007d31c000 08029 (v01 DELL PE_SC3 00000001 INTL 20110211) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: FACS 000000007d337000 00040 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: APIC 000000007d334478 001EA (v01 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: SPCR 000000007d334664 00050 (v01 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: HPET 000000007d3346b8 00038 (v01 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: DMAR 000000007d3346f4 00138 (v01 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: MCFG 000000007d334a10 0003C (v01 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: WD__ 000000007d334a50 00134 (v01 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: SLIC 000000007d334b88 00024 (v01 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: ERST 000000007d3241ec 00270 (v01 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: HEST 000000007d32445c 0055C (v01 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: BERT 000000007d32402c 00030 (v01 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: EINJ 000000007d32405c 00190 (v01 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: TCPA 000000007d3352e4 00064 (v02 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: PC__ 000000007d335274 0006E (v01 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: SRAT 000000007d334db0 004C0 (v01 DELL PE_SC3 00000001 DELL 00000001) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: SSDT 000000007d338000 0709C (v01 INTEL PPM RCM 80000001 INTL 20061109) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: Local APIC address 0xfee00000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Setting APIC routing to flat. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 1 -> APIC 0 -> Node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 2 -> APIC 32 -> Node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 1 -> APIC 2 -> Node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 2 -> APIC 34 -> Node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 1 -> APIC 4 -> Node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 2 -> APIC 36 -> Node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 1 -> APIC 6 -> Node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 2 -> APIC 38 -> Node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 1 -> APIC 8 -> Node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 2 -> APIC 40 -> Node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 1 -> APIC 10 -> Node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 2 -> APIC 42 -> Node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 1 -> APIC 1 -> Node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 2 -> APIC 33 -> Node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 1 -> APIC 3 -> Node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 2 -> APIC 35 -> Node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 1 -> APIC 5 -> Node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 2 -> APIC 37 -> Node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 1 -> APIC 7 -> Node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 2 -> APIC 39 -> Node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 1 -> APIC 9 -> Node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 2 -> APIC 41 -> Node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 1 -> APIC 11 -> Node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: PXM 2 -> APIC 43 -> Node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: Node 0 PXM 1 0-2080000000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SRAT: Node 1 PXM 2 2080000000-4080000000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] NUMA: Allocated memnodemap from a000 - a140 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] NUMA: Using 31 for the hash shift. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Bootmem setup node 0 0000000000000000-0000002080000000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] NODE_DATA [000000000000a140 - 000000000003e13f] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] bootmap [0000000000100000 - 000000000050ffff] pages 410 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] (8 early reservations) ==> bootmem [0000000000 - 2080000000] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #2 [0001000000 - 000201b0e4] TEXT DATA BSS ==> [0001000000 - 000201b0e4] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #3 [007c594000 - 007d2cf2df] RAMDISK ==> [007c594000 - 007d2cf2df] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #4 [000009d800 - 0000100000] BIOS reserved ==> [000009d800 - 0000100000] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #5 [000201c000 - 000201c675] BRK ==> [000201c000 - 000201c675] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #6 [0000008000 - 000000a000] PGTABLE ==> [0000008000 - 000000a000] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #7 [000000a000 - 000000a140] MEMNODEMAP ==> [000000a000 - 000000a140] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Bootmem setup node 1 0000002080000000-0000004080000000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] NODE_DATA [0000002080000040 - 000000208003403f] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] bootmap [0000002080035000 - 0000002080434fff] pages 400 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] (8 early reservations) ==> bootmem [2080000000 - 4080000000] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #0 [0000000000 - 0000001000] BIOS data page Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #1 [0000006000 - 0000008000] TRAMPOLINE Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #2 [0001000000 - 000201b0e4] TEXT DATA BSS Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #3 [007c594000 - 007d2cf2df] RAMDISK Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #4 [000009d800 - 0000100000] BIOS reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #5 [000201c000 - 000201c675] BRK Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #6 [0000008000 - 000000a000] PGTABLE Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] #7 [000000a000 - 000000a140] MEMNODEMAP Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] found SMP MP-table at [ffff8800000fe710] fe710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Reserving 192MB of memory at 48MB for crashkernel (System RAM: 264192MB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] [ffffea0000000000-ffffea003fffffff] PMD -> [ffff880028600000-ffff8800669fffff] on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] [ffffea0040000000-ffffea00557fffff] PMD -> [ffff880066c00000-ffff88007c3fffff] on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] [ffffea0055800000-ffffea0071bfffff] PMD -> [ffff880100000000-ffff88011c3fffff] on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] [ffffea0071c00000-ffffea007fffffff] PMD -> [ffff882080600000-ffff88208e9fffff] on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] [ffffea0080000000-ffffea00bfffffff] PMD -> [ffff88208ec00000-ffff8820cebfffff] on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] [ffffea00c0000000-ffffea00e1bfffff] PMD -> [ffff8820cee00000-ffff8820f09fffff] on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Zone PFN ranges: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] DMA 0x00000001 -> 0x00001000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] DMA32 0x00001000 -> 0x00100000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Normal 0x00100000 -> 0x04080000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Movable zone start PFN for each node Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] early_node_map[4] active PFN ranges Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 0: 0x00000001 -> 0x0000009d Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 0: 0x00000100 -> 0x0007d2f0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 0: 0x00100000 -> 0x02080000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] 1: 0x02080000 -> 0x04080000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] On node 0 totalpages: 33542796 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] DMA zone: 56 pages used for memmap Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] DMA zone: 103 pages reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] DMA zone: 3837 pages, LIFO batch:0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] DMA32 zone: 14280 pages used for memmap Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] DMA32 zone: 494376 pages, LIFO batch:31 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Normal zone: 451584 pages used for memmap Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Normal zone: 32578560 pages, LIFO batch:31 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] On node 1 totalpages: 33554432 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Normal zone: 458752 pages used for memmap Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Normal zone: 33095680 pages, LIFO batch:31 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: PM-Timer IO Port: 0x808 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: Local APIC address 0xfee00000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Setting APIC routing to flat. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x20] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x22] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x24] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x06] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x26] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x08] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x28] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0a] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x2a] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x01] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x21] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x03] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x23] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0x05] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0x25] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0x07] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0x27] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x15] lapic_id[0x09] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x16] lapic_id[0x29] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x17] lapic_id[0x0b] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x18] lapic_id[0x2b] enabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x19] lapic_id[0x58] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x59] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x5a] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x5b] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x5c] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x5d] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x5e] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x20] lapic_id[0x5f] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x21] lapic_id[0x60] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x22] lapic_id[0x61] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x23] lapic_id[0x62] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x24] lapic_id[0x63] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x25] lapic_id[0x64] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x26] lapic_id[0x65] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x27] lapic_id[0x66] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x28] lapic_id[0x67] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x29] lapic_id[0x68] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x2a] lapic_id[0x69] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x2b] lapic_id[0x6a] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x2c] lapic_id[0x6b] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x2d] lapic_id[0x6c] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x2e] lapic_id[0x6d] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x2f] lapic_id[0x6e] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC (acpi_id[0x30] lapic_id[0x6f] disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[32]) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 32-55 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[64]) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 64-87 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: IRQ0 used by override. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: IRQ2 used by override. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: IRQ9 used by override. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Using ACPI (MADT) for SMP configuration information Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] SMP: Allowing 48 CPUs, 24 hotplug CPUs Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] nr_irqs_gsi: 88 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] PM: Registered nosave memory: 000000000009d000 - 0000000000100000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] PM: Registered nosave memory: 000000007d2f0000 - 000000007d31c000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] PM: Registered nosave memory: 000000007d31c000 - 000000007d35b000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] PM: Registered nosave memory: 000000007d35b000 - 0000000080000000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] PM: Registered nosave memory: 0000000080000000 - 00000000e0000000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] PM: Registered nosave memory: 00000000e0000000 - 00000000f0000000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] PM: Registered nosave memory: 00000000f0000000 - 00000000fe000000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] PM: Registered nosave memory: 00000000fe000000 - 0000000100000000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Allocating PCI resources starting at 80000000 (gap: 80000000:60000000) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Booting paravirtualized kernel on bare hardware Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] NR_CPUS:4096 nr_cpumask_bits:48 nr_cpu_ids:48 nr_node_ids:2 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] PERCPU: Embedded 31 pages/cpu @ffff88011c400000 s94616 r8192 d24168 u131072 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] pcpu-alloc: s94616 r8192 d24168 u131072 alloc=1*2097152 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] pcpu-alloc: [0] 00 02 04 06 08 10 12 14 16 18 20 22 24 26 28 30 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] pcpu-alloc: [0] 32 34 36 38 40 42 44 46 -- -- -- -- -- -- -- -- Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] pcpu-alloc: [1] 01 03 05 07 09 11 13 15 17 19 21 23 25 27 29 31 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] pcpu-alloc: [1] 33 35 37 39 41 43 45 47 -- -- -- -- -- -- -- -- Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Built 2 zonelists in Zone order, mobility grouping on. Total pages: 66172453 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Policy zone: Normal Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Kernel command line: initrd=initrd-2.6.32-358.23.2.el6.atlas.x86_64-gedi selinux=0 audit=0 panic=10 console=tty0 console=ttyS1,115200n8 console=ttyS2,115200n8 crashkernel=192M BOOT_IMAGE=vmlinuz-2.6.32-358.23.2.el6.atlas.x86_64 BOOTIF=01-90-b1-1c-38-49-d3 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] audit: disabled (until reboot) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Checking aperture... Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] No AGP bridge found Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] software IO TLB at phys 0x20000000 - 0x24000000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Memory: 264419192k/270532608k available (5216k kernel code, 2143696k absent, 3969720k reserved, 7126k data, 1264k init) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Hierarchical RCU implementation. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] NR_IRQS:33024 nr_irqs:1880 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Extended CMOS year: 2000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Console: colour VGA+ 80x25 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] console [tty0] enabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] console [ttyS1] enabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] allocated 1073741824 bytes of page_cgroup Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] hpet clockevent registered Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000000] Fast TSC calibration using PIT Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.001000] Detected 2500.024 MHz processor. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.000014] Calibrating delay loop (skipped), value calculated using timer frequency.. 5000.04 BogoMIPS (lpj=2500024) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.012263] pid_max: default: 49152 minimum: 384 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.017800] Security Framework initialized Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.022565] SELinux: Disabled at boot. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.049184] Dentry cache hash table entries: 33554432 (order: 16, 268435456 bytes) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.107823] Inode-cache hash table entries: 16777216 (order: 15, 134217728 bytes) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.136516] Mount-cache hash table entries: 256 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.141957] Initializing cgroup subsys ns Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.146626] Initializing cgroup subsys cpuacct Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.151781] Initializing cgroup subsys memory Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.156855] Initializing cgroup subsys devices Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.162010] Initializing cgroup subsys freezer Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.167160] Initializing cgroup subsys net_cls Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.172309] Initializing cgroup subsys blkio Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.177268] Initializing cgroup subsys perf_event Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.182712] Initializing cgroup subsys net_prio Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.188004] CPU: Physical Processor ID: 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.192667] CPU: Processor Core ID: 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.196975] mce: CPU supports 18 MCE banks Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.201759] CPU0: Thermal monitoring enabled (TM1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.207315] using mwait in idle threads. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.213169] ACPI: Core revision 20090903 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.221307] ftrace: converting mcount calls to 0f 1f 44 00 00 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.227918] ftrace: allocating 21440 entries in 85 pages Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.241509] dmar: Host address width 46 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.245986] dmar: DRHD base: 0x000000d5100000 flags: 0x0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.252119] dmar: IOMMU 0: reg_base_addr d5100000 ver 1:0 cap d2078c106f0462 ecap f020fe Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.261452] dmar: DRHD base: 0x000000de900000 flags: 0x1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.267582] dmar: IOMMU 1: reg_base_addr de900000 ver 1:0 cap d2078c106f0462 ecap f020fe Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.276910] dmar: RMRR base: 0x0000007f458000 end: 0x0000007f46ffff Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.284101] dmar: RMRR base: 0x0000007f450000 end: 0x0000007f450fff Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.291294] dmar: RMRR base: 0x0000007f452000 end: 0x0000007f452fff Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.298485] dmar: ATSR flags: 0x0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.302375] IOAPIC id 2 under DRHD base 0xd5100000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.307907] IOAPIC id 0 under DRHD base 0xde900000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.313441] IOAPIC id 1 under DRHD base 0xde900000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.318980] HPET id 0 under DRHD base 0xde900000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.324314] Your BIOS has requested that x2apic be disabled. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.324315] This will leave your machine vulnerable to irq-injection attacks. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.324315] Use 'intremap=no_x2apic_optout' to override BIOS request. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.346809] Enabled IRQ remapping in xapic mode Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.352064] x2apic not enabled, IRQ remapping is in xapic mode Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.360747] APIC routing finalized to physical flat. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366910] alloc irq_desc for 88 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366911] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366922] alloc irq_desc for 89 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366923] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366931] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366936] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366941] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366945] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366950] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366954] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366959] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366963] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366968] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366972] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366977] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366981] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366986] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366990] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.366995] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.367139] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.384046] CPU0: Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz stepping 07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.493360] Performance Events: PEBS fmt1+, SandyBridge events, Intel PMU driver. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.502277] PEBS disabled due to CPU errata. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.507229] ... version: 3 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.511887] ... bit width: 48 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.516647] ... generic registers: 4 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.521305] ... value mask: 0000ffffffffffff Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.527422] ... max period: 000000007fffffff Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.533544] ... fixed-purpose events: 3 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.538199] ... event mask: 000000070000000f Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.544893] NMI watchdog enabled, takes one hw-pmu counter. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.551466] Booting Node 1, Processors #1 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.648441] Booting Node 0, Processors #2 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.686244] Booting Node 1, Processors #3 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.723141] Booting Node 0, Processors #4 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.760142] Booting Node 1, Processors #5 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.797048] Booting Node 0, Processors #6 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.834031] Booting Node 1, Processors #7 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.870935] Booting Node 0, Processors #8 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.907908] Booting Node 1, Processors #9 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.944796] Booting Node 0, Processors #10 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 0.981872] Booting Node 1, Processors #11 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.018870] Booting Node 0, Processors #12 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.056153] Booting Node 1, Processors #13 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.093154] Booting Node 0, Processors #14 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.130244] Booting Node 1, Processors #15 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.167250] Booting Node 0, Processors #16 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.204331] Booting Node 1, Processors #17 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.241336] Booting Node 0, Processors #18 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.278465] Booting Node 1, Processors #19 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.315461] Booting Node 0, Processors #20 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.352547] Booting Node 1, Processors #21 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.389539] Booting Node 0, Processors #22 Ok. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.426629] Booting Node 1, Processors #23 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.462787] Brought up 24 CPUs Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.466763] Total of 24 processors activated (119991.98 BogoMIPS). Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.497037] sizeof(vma)=200 bytes Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.497039] sizeof(page)=56 bytes Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.497040] sizeof(inode)=592 bytes Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.497041] sizeof(dentry)=192 bytes Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.497042] sizeof(ext3inode)=800 bytes Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.497042] sizeof(buffer_head)=104 bytes Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.497043] sizeof(skbuff)=232 bytes Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.497044] sizeof(task_struct)=2648 bytes Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.497930] devtmpfs: initialized Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.546277] regulator: core version 0.5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.550868] NET: Registered protocol family 16 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.556815] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.565558] ACPI: bus type pci registered Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.570287] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.578356] PCI: MCFG area at e0000000 reserved in E820 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.606053] PCI: Using MMCONFIG at e0000000 - efffffff Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.611976] PCI: Using configuration type 1 for base access Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.618717] PCI: Dell System detected, enabling pci=bfsort. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.629486] bio: create slab at 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.635309] ACPI: EC: Look up EC in DSDT Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.637403] ACPI: BIOS _OSI(Linux) query ignored Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.643252] ACPI: Interpreter enabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.647547] ACPI: (supports S0 S4 S5) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.654617] ACPI: Using IOAPIC for interrupt routing Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.664387] ACPI: No dock devices found. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.669033] HEST: Table parsing has been initialized. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.674861] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.688984] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-3d]) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.697535] pci_root PNP0A08:00: host bridge window [io 0x0000-0x03af] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.705113] pci_root PNP0A08:00: host bridge window [io 0x03e0-0x0cf7] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.712691] pci_root PNP0A08:00: host bridge window [io 0x03b0-0x03df] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.720268] pci_root PNP0A08:00: host bridge window [io 0x0d00-0x1fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.727847] pci_root PNP0A08:00: host bridge window [io 0xe000-0xffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.735426] pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.743876] pci_root PNP0A08:00: host bridge window [mem 0xd8000000-0xdfffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.752327] pci_root PNP0A08:00: host bridge window [mem 0xfed40000-0xfed44fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.760837] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.767833] pci 0000:00:00.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.772750] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.779741] pci 0000:00:01.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.784659] pci 0000:00:01.1: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.791652] pci 0000:00:01.1: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.796572] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.803570] pci 0000:00:02.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.808488] pci 0000:00:02.2: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.815482] pci 0000:00:02.2: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.820398] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.827385] pci 0000:00:03.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.832445] pci 0000:00:11.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.839444] pci 0000:00:11.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.844330] pci 0000:00:16.0: reg 10: [mem 0xde8fb000-0xde8fb00f 64bit] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.844380] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.851373] pci 0000:00:16.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.856258] pci 0000:00:16.1: reg 10: [mem 0xde8fc000-0xde8fc00f 64bit] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.856307] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.863304] pci 0000:00:16.1: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.868205] pci 0000:00:1a.0: reg 10: [mem 0xde8fd000-0xde8fd3ff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.868275] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.875275] pci 0000:00:1a.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.880195] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.887192] pci 0000:00:1c.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.892115] pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.899113] pci 0000:00:1c.7: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.904011] pci 0000:00:1d.0: reg 10: [mem 0xde8fe000-0xde8fe3ff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.904081] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.911079] pci 0000:00:1d.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.916123] pci 0000:00:1f.2: reg 10: [io 0xfce8-0xfcef] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.916130] pci 0000:00:1f.2: reg 14: [io 0xfcf8-0xfcfb] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.916137] pci 0000:00:1f.2: reg 18: [io 0xfcf0-0xfcf7] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.916144] pci 0000:00:1f.2: reg 1c: [io 0xfcfc-0xfcff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.916151] pci 0000:00:1f.2: reg 20: [io 0xfcc0-0xfcdf] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.916157] pci 0000:00:1f.2: reg 24: [mem 0xde8ff000-0xde8ff7ff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.916188] pci 0000:00:1f.2: PME# supported from D3hot Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.922208] pci 0000:00:1f.2: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.927105] pci 0000:02:00.0: reg 10: [mem 0xd90a0000-0xd90affff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.927116] pci 0000:02:00.0: reg 18: [mem 0xd90b0000-0xd90bffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.927127] pci 0000:02:00.0: reg 20: [mem 0xd90c0000-0xd90cffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.927134] pci 0000:02:00.0: reg 30: [mem 0xdc000000-0xdc03ffff pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.927161] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.934159] pci 0000:02:00.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.939052] pci 0000:02:00.1: reg 10: [mem 0xd90d0000-0xd90dffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.939063] pci 0000:02:00.1: reg 18: [mem 0xd90e0000-0xd90effff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.939073] pci 0000:02:00.1: reg 20: [mem 0xd90f0000-0xd90fffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.939081] pci 0000:02:00.1: reg 30: [mem 0xdc000000-0xdc03ffff pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.939107] pci 0000:02:00.1: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.946104] pci 0000:02:00.1: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.951032] pci 0000:00:01.0: PCI bridge to [bus 02-02] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.957057] pci 0000:00:01.0: bridge window [io 0xf000-0x0000] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.957060] pci 0000:00:01.0: bridge window [mem 0xdc000000-0xdc7fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.957064] pci 0000:00:01.0: bridge window [mem 0xd9000000-0xd90fffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.957097] pci 0000:01:00.0: reg 10: [mem 0xd91a0000-0xd91affff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.957107] pci 0000:01:00.0: reg 18: [mem 0xd91b0000-0xd91bffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.957118] pci 0000:01:00.0: reg 20: [mem 0xd91c0000-0xd91cffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.957125] pci 0000:01:00.0: reg 30: [mem 0xdc800000-0xdc83ffff pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.957152] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.964149] pci 0000:01:00.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.969040] pci 0000:01:00.1: reg 10: [mem 0xd91d0000-0xd91dffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.969051] pci 0000:01:00.1: reg 18: [mem 0xd91e0000-0xd91effff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.969062] pci 0000:01:00.1: reg 20: [mem 0xd91f0000-0xd91fffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.969069] pci 0000:01:00.1: reg 30: [mem 0xdc800000-0xdc83ffff pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.969096] pci 0000:01:00.1: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.976093] pci 0000:01:00.1: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.981023] pci 0000:00:01.1: PCI bridge to [bus 01-01] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.987041] pci 0000:00:01.1: bridge window [io 0xf000-0x0000] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.987044] pci 0000:00:01.1: bridge window [mem 0xdc800000-0xdcffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.987049] pci 0000:00:01.1: bridge window [mem 0xd9100000-0xd91fffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.987066] pci 0000:00:02.0: PCI bridge to [bus 04-04] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.993085] pci 0000:00:02.0: bridge window [io 0xf000-0x0000] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.993088] pci 0000:00:02.0: bridge window [mem 0xfff00000-0x000fffff] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.993093] pci 0000:00:02.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.993110] pci 0000:00:02.2: PCI bridge to [bus 03-03] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.999125] pci 0000:00:02.2: bridge window [io 0xf000-0x0000] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.999127] pci 0000:00:02.2: bridge window [mem 0xfff00000-0x000fffff] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.999132] pci 0000:00:02.2: bridge window [mem 0xfff00000-0x000fffff pref] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 1.999149] pci 0000:00:03.0: PCI bridge to [bus 05-05] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.005173] pci 0000:00:03.0: bridge window [io 0xf000-0x0000] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.005176] pci 0000:00:03.0: bridge window [mem 0xfff00000-0x000fffff] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.005180] pci 0000:00:03.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.005216] pci 0000:00:11.0: PCI bridge to [bus 06-06] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.011239] pci 0000:00:11.0: bridge window [io 0xf000-0x0000] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.011242] pci 0000:00:11.0: bridge window [mem 0xfff00000-0x000fffff] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.011248] pci 0000:00:11.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.011282] pci 0000:00:1c.0: PCI bridge to [bus 07-07] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.017310] pci 0000:00:1c.0: bridge window [io 0xf000-0x0000] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.017313] pci 0000:00:1c.0: bridge window [mem 0xfff00000-0x000fffff] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.017319] pci 0000:00:1c.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.019221] pci 0000:08:00.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.026295] pci 0000:08:00.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.032032] pci 0000:00:1c.7: PCI bridge to [bus 08-0c] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.038056] pci 0000:00:1c.7: bridge window [io 0xf000-0x0000] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.038059] pci 0000:00:1c.7: bridge window [mem 0xdd000000-0xde7fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.038065] pci 0000:00:1c.7: bridge window [mem 0xd8000000-0xd8ffffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.040411] pci 0000:09:00.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.047483] pci 0000:09:00.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.054891] pci 0000:09:01.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.061964] pci 0000:09:01.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.068811] pci 0000:08:00.0: PCI bridge to [bus 09-0c] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.074967] pci 0000:08:00.0: bridge window [io 0xf000-0x0000] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.075031] pci 0000:08:00.0: bridge window [mem 0xdd000000-0xde7fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.075155] pci 0000:08:00.0: bridge window [mem 0xd8000000-0xd8ffffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.080478] pci 0000:09:00.0: PCI bridge to [bus 0a-0b] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.086649] pci 0000:09:00.0: bridge window [io 0xf000-0x0000] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.086722] pci 0000:09:00.0: bridge window [mem 0xdd000000-0xddffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.086866] pci 0000:09:00.0: bridge window [mem 0xd8000000-0xd8ffffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.087697] pci 0000:0b:00.0: reg 10: [mem 0xd8000000-0xd8ffffff pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.087957] pci 0000:0b:00.0: reg 14: [mem 0xddffc000-0xddffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.088219] pci 0000:0b:00.0: reg 18: [mem 0xdd000000-0xdd7fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.092004] pci 0000:0a:00.0: PCI bridge to [bus 0b-0b] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.098176] pci 0000:0a:00.0: bridge window [io 0xf000-0x0000] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.098249] pci 0000:0a:00.0: bridge window [mem 0xdd000000-0xddffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.098394] pci 0000:0a:00.0: bridge window [mem 0xd8000000-0xd8ffffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.100528] pci 0000:09:01.0: PCI bridge to [bus 0c-0c] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.106693] pci 0000:09:01.0: bridge window [io 0xf000-0x0000] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.106765] pci 0000:09:01.0: bridge window [mem 0xde700000-0xde7fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.106910] pci 0000:09:01.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.107922] pci 0000:00:1e.0: PCI bridge to [bus 0d-0d] (subtractive decode) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.115995] pci 0000:00:1e.0: bridge window [io 0xf000-0x0000] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.115998] pci 0000:00:1e.0: bridge window [mem 0xfff00000-0x000fffff] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116002] pci 0000:00:1e.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116004] pci 0000:00:1e.0: bridge window [io 0x0000-0x03af] (subtractive decode) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116006] pci 0000:00:1e.0: bridge window [io 0x03e0-0x0cf7] (subtractive decode) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116007] pci 0000:00:1e.0: bridge window [io 0x03b0-0x03df] (subtractive decode) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116008] pci 0000:00:1e.0: bridge window [io 0x0d00-0x1fff] (subtractive decode) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116010] pci 0000:00:1e.0: bridge window [io 0xe000-0xffff] (subtractive decode) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116011] pci 0000:00:1e.0: bridge window [mem 0x000a0000-0x000bffff] (subtractive decode) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116013] pci 0000:00:1e.0: bridge window [mem 0xd8000000-0xdfffffff] (subtractive decode) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116014] pci 0000:00:1e.0: bridge window [mem 0xfed40000-0xfed44fff] (subtractive decode) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116049] pci_bus 0000:00: on NUMA node 0 (pxm 1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116052] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116406] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX1._PRT] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116519] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PE1C._PRT] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116610] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX2._PRT] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116693] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX3._PRT] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116774] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX4._PRT] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.116857] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX6._PRT] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.117077] pci0000:00: Requesting ACPI _OSC control (0x15) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.123832] pci0000:00: ACPI _OSC control (0x15) granted Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.135498] ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 40-7e]) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.144147] pci_root PNP0A08:01: host bridge window [io 0x2000-0x3fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.151724] pci_root PNP0A08:01: host bridge window [io 0x4000-0x5fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.159307] pci_root PNP0A08:01: host bridge window [io 0x6000-0x7fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.166885] pci_root PNP0A08:01: host bridge window [io 0x8000-0x9fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.174465] pci_root PNP0A08:01: host bridge window [io 0xa000-0xbfff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.182043] pci_root PNP0A08:01: host bridge window [io 0xc000-0xdfff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.189621] pci_root PNP0A08:01: host bridge window [mem 0xd0000000-0xd7ffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.198135] pci 0000:40:01.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.205131] pci 0000:40:01.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.210055] pci 0000:40:03.0: PME# supported from D0 D3hot D3cold Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.217050] pci 0000:40:03.0: PME# disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.222079] pci 0000:41:00.0: reg 10: [io 0xdc00-0xdcff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.222087] pci 0000:41:00.0: reg 14: [mem 0xd4ff0000-0xd4ffffff 64bit] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.222096] pci 0000:41:00.0: reg 1c: [mem 0xd4f80000-0xd4fbffff 64bit] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.222106] pci 0000:41:00.0: reg 30: [mem 0xd4000000-0xd40fffff pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.222122] pci 0000:41:00.0: supports D1 D2 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.222160] pci 0000:40:01.0: PCI bridge to [bus 41-41] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.228184] pci 0000:40:01.0: bridge window [io 0xd000-0xdfff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.228187] pci 0000:40:01.0: bridge window [mem 0xd4000000-0xd4ffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.228192] pci 0000:40:01.0: bridge window [mem 0xfff00000-0x000fffff pref] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.228638] pci 0000:42:00.0: reg 10: [mem 0xd5000000-0xd50fffff 64bit] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.228844] pci 0000:42:00.0: reg 18: [mem 0xd0000000-0xd07fffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.230259] pci 0000:40:03.0: PCI bridge to [bus 42-42] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.236281] pci 0000:40:03.0: bridge window [io 0xf000-0x0000] (disabled) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.236284] pci 0000:40:03.0: bridge window [mem 0xd5000000-0xd50fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.236288] pci 0000:40:03.0: bridge window [mem 0xd0000000-0xd07fffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.236298] pci_bus 0000:40: on NUMA node 1 (pxm 2) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.236300] ACPI: PCI Interrupt Routing Table [\_SB_.PCI1._PRT] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.236451] ACPI: PCI Interrupt Routing Table [\_SB_.PCI1.PEXB._PRT] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.236536] ACPI: PCI Interrupt Routing Table [\_SB_.PCI1.PEXD._PRT] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.236745] pci0000:40: Requesting ACPI _OSC control (0x15) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.243497] pci0000:40: ACPI _OSC control (0x15) granted Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.250316] ACPI: PCI Root Bridge [P0B1] (domain 0000 [bus 3f]) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.258269] pci_bus 0000:3f: on NUMA node 0 (pxm 1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.258399] pci0000:3f: Requesting ACPI _OSC control (0x15) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.265144] pci0000:3f: ACPI _OSC control (0x15) granted Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.274483] ACPI: PCI Root Bridge [P1B1] (domain 0000 [bus 7f]) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.282578] pci_bus 0000:7f: on NUMA node 1 (pxm 2) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.282705] pci0000:7f: Requesting ACPI _OSC control (0x15) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.289456] pci0000:7f: ACPI _OSC control (0x15) granted Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.296381] ACPI: PCI Interrupt Link [LK00] (IRQs 3 4 5 6 7 11 14 *15) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.304819] ACPI: PCI Interrupt Link [LK01] (IRQs 3 4 5 6 7 11 *14 15) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.313251] ACPI: PCI Interrupt Link [LK02] (IRQs 3 4 5 6 7 11 14 15) *0, disabled. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.323223] ACPI: PCI Interrupt Link [LK03] (IRQs 3 4 5 6 7 *11 14 15) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.331653] ACPI: PCI Interrupt Link [LK04] (IRQs 3 4 *5 6 7 11 14 15) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.340075] ACPI: PCI Interrupt Link [LK05] (IRQs 3 4 5 6 7 11 14 15) *0, disabled. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.350043] ACPI: PCI Interrupt Link [LK06] (IRQs 3 4 5 *6 7 11 14 15) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.358464] ACPI: PCI Interrupt Link [LK07] (IRQs 3 4 5 6 7 11 *14 15) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.367051] vgaarb: device added: PCI:0000:0b:00.0,decodes=io+mem,owns=io+mem,locks=none Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.376413] vgaarb: loaded Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.379613] vgaarb: bridge control possible 0000:0b:00.0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.386022] SCSI subsystem initialized Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.390822] libata version 3.00 loaded. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.390899] usbcore: registered new interface driver usbfs Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.397229] usbcore: registered new interface driver hub Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.403375] usbcore: registered new device driver usb Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.410067] PCI: Using ACPI for IRQ routing Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.414925] PCI: old code would have set cacheline size to 32 bytes, but clflush_size = 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.414926] PCI: pci_cache_line_size set to 64 bytes Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.415655] NetLabel: Initializing Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.419633] NetLabel: domain hash size = 128 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.424681] NetLabel: protocols = UNLABELED CIPSOv4 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.430422] NetLabel: unlabeled traffic allowed by default Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436910] alloc irq_desc for 90 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436912] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436916] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436921] alloc irq_desc for 91 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436922] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436925] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436929] alloc irq_desc for 92 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436930] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436932] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436936] alloc irq_desc for 93 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436937] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436939] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436943] alloc irq_desc for 94 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436944] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436947] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.436951] HPET: 8 timers in total, 5 timers will be used for per-cpu timer Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.445024] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 90, 91, 92, 93, 94, 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.453412] hpet0: 8 comparators, 64-bit 14.318180 MHz counter Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.462987] hpet: hpet2 irq 90 for MSI Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.463089] hpet: hpet3 irq 91 for MSI Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.463750] hpet: hpet4 irq 92 for MSI Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.464718] hpet: hpet5 irq 93 for MSI Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.465592] hpet: hpet6 irq 94 for MSI Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.475638] Switching to clocksource hpet Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.488456] pnp: PnP ACPI init Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.492063] ACPI: bus type pnp registered Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499376] pnp 00:00: [io 0x0cf8-0x0cff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499419] pnp 00:00: Plug and Play ACPI device, IDs PNP0a08 PNP0a03 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499564] pnp 00:01: [io 0x0080-0x009f] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499566] pnp 00:01: [io 0x0000-0x001f] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499567] pnp 00:01: [io 0x00c0-0x00df] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499569] pnp 00:01: [dma 4] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499599] pnp 00:01: Plug and Play ACPI device, IDs PNP0200 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499619] pnp 00:02: [io 0x00f0-0x00ff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499627] pnp 00:02: [irq 13] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499657] pnp 00:02: Plug and Play ACPI device, IDs PNP0c04 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499679] pnp 00:03: [io 0x0061] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499759] pnp 00:03: Plug and Play ACPI device, IDs PNP0c01 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499778] pnp 00:04: [io 0x0070-0x007f] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499785] pnp 00:04: [irq 8] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499812] pnp 00:04: Plug and Play ACPI device, IDs PNP0b00 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499962] pnp 00:05: [io 0x03f8-0x03ff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.499968] pnp 00:05: [irq 4] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500034] pnp 00:05: Plug and Play ACPI device, IDs PNP0501 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500177] pnp 00:06: [io 0x02f8-0x02ff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500183] pnp 00:06: [irq 3] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500245] pnp 00:06: Plug and Play ACPI device, IDs PNP0501 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500578] pnp 00:07: [io 0x0800-0x087f] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500579] pnp 00:07: [io 0x00e0-0x00e7] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500581] pnp 00:07: [io 0x0880-0x08ff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500582] pnp 00:07: [io 0x0900-0x091f] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500583] pnp 00:07: [io 0x0920-0x0923] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500585] pnp 00:07: [io 0x0924] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500587] pnp 00:07: [io 0x0370-0x0377] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500588] pnp 00:07: [io 0x0ca0-0x0ca7] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500590] pnp 00:07: [io 0x0ca9-0x0cab] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500591] pnp 00:07: [io 0x0cad-0x0caf] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500592] pnp 00:07: [io 0x0cb0-0x0cbf] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500594] pnp 00:07: [io 0x0060-0x005f disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500595] pnp 00:07: [io 0x0064-0x0063 disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500680] pnp 00:07: Plug and Play ACPI device, IDs PNP0c01 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500727] pnp 00:08: [io 0x0ca8] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500728] pnp 00:08: [io 0x0cac] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500734] pnp 00:08: [irq 10] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.500815] pnp 00:08: Plug and Play ACPI device, IDs IPI0001 PNP0c01 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.501153] pnp 00:09: [mem 0xe0000000-0xe3efffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.501235] pnp 00:09: Plug and Play ACPI device, IDs PNP0c02 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.502067] pnp 00:0a: Plug and Play ACPI device, IDs PNP0a08 PNP0a03 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.502176] pnp 00:0b: [mem 0xe4000000-0xe7ffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.502260] pnp 00:0b: Plug and Play ACPI device, IDs PNP0c02 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.502322] pnp 00:0c: Plug and Play ACPI device, IDs PNP0a08 PNP0a03 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.502343] pnp 00:0d: [mem 0xe3f00000-0xe3ffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.502427] pnp 00:0d: Plug and Play ACPI device, IDs PNP0c02 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.502458] pnp 00:0e: [mem 0xde900000-0xde901fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.502547] pnp 00:0e: Plug and Play ACPI device, IDs PNP0c02 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.502599] pnp 00:0f: [mem 0xd5100000-0xd5101fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.502684] pnp 00:0f: Plug and Play ACPI device, IDs PNP0c02 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.502733] pnp 00:10: Plug and Play ACPI device, IDs PNP0a08 PNP0a03 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.502775] pnp 00:11: [mem 0xfed00000-0xfed003ff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.502805] pnp 00:11: Plug and Play ACPI device, IDs PNP0103 (active) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.502973] pnp: PnP ACPI: found 18 devices Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.507832] ACPI: ACPI bus type pnp unregistered Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.513188] system 00:07: [io 0x0800-0x087f] has been reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.519991] system 00:07: [io 0x0880-0x08ff] has been reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.526793] system 00:07: [io 0x0900-0x091f] has been reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.533594] system 00:07: [io 0x0920-0x0923] has been reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.540395] system 00:07: [io 0x0924] has been reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.546515] system 00:07: [io 0x0370-0x0377] has been reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.553322] system 00:07: [io 0x0ca0-0x0ca7] has been reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.560126] system 00:07: [io 0x0ca9-0x0cab] has been reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.566931] system 00:07: [io 0x0cad-0x0caf] has been reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.573733] system 00:07: [io 0x0cb0-0x0cbf] has been reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.580539] system 00:08: [io 0x0ca8] has been reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.586665] system 00:08: [io 0x0cac] has been reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.592790] system 00:09: [mem 0xe0000000-0xe3efffff] has been reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.600375] system 00:0b: [mem 0xe4000000-0xe7ffffff] has been reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.607957] system 00:0d: [mem 0xe3f00000-0xe3ffffff] has been reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.615538] system 00:0e: [mem 0xde900000-0xde901fff] could not be reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.623511] system 00:0f: [mem 0xd5100000-0xd5101fff] could not be reserved Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.636151] pci 0000:01:00.1: address space collision: [mem 0xdc800000-0xdc83ffff pref] conflicts with 0000:01:00.0 [mem 0xdc800000-0xdc83ffff pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.651324] pci 0000:02:00.1: address space collision: [mem 0xdc000000-0xdc03ffff pref] conflicts with 0000:02:00.0 [mem 0xdc000000-0xdc03ffff pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.666527] PCI: max bus depth: 4 pci_try_num: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.667526] pci 0000:02:00.1: BAR 6: assigned [mem 0xd9000000-0xd903ffff pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.675884] pci 0000:00:01.0: PCI bridge to [bus 02-02] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.681907] pci 0000:00:01.0: PCI bridge to [bus 02-02] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.687931] pci 0000:00:01.0: bridge window [io disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.694443] pci 0000:00:01.0: bridge window [mem 0xdc000000-0xdc7fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.702224] pci 0000:00:01.0: bridge window [mem 0xd9000000-0xd90fffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.711162] pci 0000:01:00.1: BAR 6: assigned [mem 0xd9100000-0xd913ffff pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.719519] pci 0000:00:01.1: PCI bridge to [bus 01-01] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.725542] pci 0000:00:01.1: PCI bridge to [bus 01-01] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.731566] pci 0000:00:01.1: bridge window [io disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.738079] pci 0000:00:01.1: bridge window [mem 0xdc800000-0xdcffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.745855] pci 0000:00:01.1: bridge window [mem 0xd9100000-0xd91fffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.754798] pci 0000:00:02.0: PCI bridge to [bus 04-04] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.760820] pci 0000:00:02.0: PCI bridge to [bus 04-04] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.766842] pci 0000:00:02.0: bridge window [io disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.773354] pci 0000:00:02.0: bridge window [mem disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.779865] pci 0000:00:02.0: bridge window [mem pref disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.786864] pci 0000:00:02.2: PCI bridge to [bus 03-03] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.792889] pci 0000:00:02.2: PCI bridge to [bus 03-03] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.798919] pci 0000:00:02.2: bridge window [io disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.805510] pci 0000:00:02.2: bridge window [mem disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.812022] pci 0000:00:02.2: bridge window [mem pref disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.819021] pci 0000:00:03.0: PCI bridge to [bus 05-05] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.825044] pci 0000:00:03.0: PCI bridge to [bus 05-05] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.831067] pci 0000:00:03.0: bridge window [io disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.837579] pci 0000:00:03.0: bridge window [mem disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.844090] pci 0000:00:03.0: bridge window [mem pref disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.851093] pci 0000:00:11.0: PCI bridge to [bus 06-06] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.857117] pci 0000:00:11.0: PCI bridge to [bus 06-06] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.863140] pci 0000:00:11.0: bridge window [io disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.869657] pci 0000:00:11.0: bridge window [mem disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.876167] pci 0000:00:11.0: bridge window [mem pref disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.883168] pci 0000:00:1c.0: PCI bridge to [bus 07-07] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.889193] pci 0000:00:1c.0: PCI bridge to [bus 07-07] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.895215] pci 0000:00:1c.0: bridge window [io disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.901729] pci 0000:00:1c.0: bridge window [mem disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.908238] pci 0000:00:1c.0: bridge window [mem pref disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.915238] pci 0000:0a:00.0: PCI bridge to [bus 0b-0b] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.921261] pci 0000:0a:00.0: PCI bridge to [bus 0b-0b] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.927289] pci 0000:0a:00.0: bridge window [io disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.933935] pci 0000:0a:00.0: bridge window [mem 0xdd000000-0xddffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.941805] pci 0000:0a:00.0: bridge window [mem 0xd8000000-0xd8ffffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.950931] pci 0000:09:00.0: PCI bridge to [bus 0a-0b] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.956963] pci 0000:09:00.0: PCI bridge to [bus 0a-0b] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.962990] pci 0000:09:00.0: bridge window [io disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.969638] pci 0000:09:00.0: bridge window [mem 0xdd000000-0xddffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.977504] pci 0000:09:00.0: bridge window [mem 0xd8000000-0xd8ffffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.986641] pci 0000:09:01.0: PCI bridge to [bus 0c-0c] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.992672] pci 0000:09:01.0: PCI bridge to [bus 0c-0c] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 2.998699] pci 0000:09:01.0: bridge window [io disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.005345] pci 0000:09:01.0: bridge window [mem 0xde700000-0xde7fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.013211] pci 0000:09:01.0: bridge window [mem pref disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.020458] pci 0000:08:00.0: PCI bridge to [bus 09-0c] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.026489] pci 0000:08:00.0: PCI bridge to [bus 09-0c] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.032508] pci 0000:08:00.0: bridge window [io disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.039148] pci 0000:08:00.0: bridge window [mem 0xdd000000-0xde7fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.047004] pci 0000:08:00.0: bridge window [mem 0xd8000000-0xd8ffffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.056186] pci 0000:00:1c.7: PCI bridge to [bus 08-0c] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.062206] pci 0000:00:1c.7: PCI bridge to [bus 08-0c] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.068231] pci 0000:00:1c.7: bridge window [io disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.074745] pci 0000:00:1c.7: bridge window [mem 0xdd000000-0xde7fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.082522] pci 0000:00:1c.7: bridge window [mem 0xd8000000-0xd8ffffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.091466] pci 0000:00:1e.0: PCI bridge to [bus 0d-0d] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.097495] pci 0000:00:1e.0: PCI bridge to [bus 0d-0d] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.103517] pci 0000:00:1e.0: bridge window [io disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.110027] pci 0000:00:1e.0: bridge window [mem disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.116538] pci 0000:00:1e.0: bridge window [mem pref disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.123536] pci 0000:40:01.0: PCI bridge to [bus 41-41] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.129559] pci 0000:40:01.0: PCI bridge to [bus 41-41] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.135583] pci 0000:40:01.0: bridge window [io 0xd000-0xdfff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.142576] pci 0000:40:01.0: bridge window [mem 0xd4000000-0xd4ffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.150350] pci 0000:40:01.0: bridge window [mem pref disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.157349] pci 0000:40:03.0: PCI bridge to [bus 42-42] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.163371] pci 0000:40:03.0: PCI bridge to [bus 42-42] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.169399] pci 0000:40:03.0: bridge window [io disabled] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.176004] pci 0000:40:03.0: bridge window [mem 0xd5000000-0xd50fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.183782] pci 0000:40:03.0: bridge window [mem 0xd0000000-0xd07fffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.192736] alloc irq_desc for 53 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.192737] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.192740] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.192745] pci 0000:00:01.0: PCI INT A -> GSI 53 (level, low) -> IRQ 53 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.200429] pci 0000:00:01.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.200444] pci 0000:00:01.1: PCI INT A -> GSI 53 (level, low) -> IRQ 53 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.208123] pci 0000:00:01.1: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.208138] pci 0000:00:02.0: PCI INT A -> GSI 53 (level, low) -> IRQ 53 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.215816] pci 0000:00:02.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.215832] pci 0000:00:02.2: PCI INT A -> GSI 53 (level, low) -> IRQ 53 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.223511] pci 0000:00:02.2: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.223527] pci 0000:00:03.0: PCI INT A -> GSI 53 (level, low) -> IRQ 53 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.231204] pci 0000:00:03.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.231217] alloc irq_desc for 16 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.231218] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.231220] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.231224] pci 0000:00:11.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.238905] pci 0000:00:11.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.238918] pci 0000:00:1c.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.238929] alloc irq_desc for 19 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.238930] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.238934] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.238937] pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.246617] pci 0000:00:1c.7: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.247096] pci 0000:08:00.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.247677] pci 0000:09:00.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.248270] pci 0000:0a:00.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.248892] pci 0000:09:01.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.248935] pci 0000:00:1e.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.248953] alloc irq_desc for 85 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.248954] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.248956] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.248961] pci 0000:40:01.0: PCI INT A -> GSI 85 (level, low) -> IRQ 85 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.256640] pci 0000:40:01.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.256659] pci 0000:40:03.0: PCI INT A -> GSI 85 (level, low) -> IRQ 85 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264340] pci 0000:40:03.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264344] pci_bus 0000:00: resource 4 [io 0x0000-0x03af] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264345] pci_bus 0000:00: resource 5 [io 0x03e0-0x0cf7] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264347] pci_bus 0000:00: resource 6 [io 0x03b0-0x03df] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264348] pci_bus 0000:00: resource 7 [io 0x0d00-0x1fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264349] pci_bus 0000:00: resource 8 [io 0xe000-0xffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264350] pci_bus 0000:00: resource 9 [mem 0x000a0000-0x000bffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264352] pci_bus 0000:00: resource 10 [mem 0xd8000000-0xdfffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264353] pci_bus 0000:00: resource 11 [mem 0xfed40000-0xfed44fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264355] pci_bus 0000:02: resource 1 [mem 0xdc000000-0xdc7fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264356] pci_bus 0000:02: resource 2 [mem 0xd9000000-0xd90fffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264358] pci_bus 0000:01: resource 1 [mem 0xdc800000-0xdcffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264359] pci_bus 0000:01: resource 2 [mem 0xd9100000-0xd91fffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264361] pci_bus 0000:08: resource 1 [mem 0xdd000000-0xde7fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264362] pci_bus 0000:08: resource 2 [mem 0xd8000000-0xd8ffffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264364] pci_bus 0000:09: resource 1 [mem 0xdd000000-0xde7fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264365] pci_bus 0000:09: resource 2 [mem 0xd8000000-0xd8ffffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264367] pci_bus 0000:0a: resource 1 [mem 0xdd000000-0xddffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264368] pci_bus 0000:0a: resource 2 [mem 0xd8000000-0xd8ffffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264370] pci_bus 0000:0b: resource 1 [mem 0xdd000000-0xddffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264371] pci_bus 0000:0b: resource 2 [mem 0xd8000000-0xd8ffffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264373] pci_bus 0000:0c: resource 1 [mem 0xde700000-0xde7fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264374] pci_bus 0000:0d: resource 4 [io 0x0000-0x03af] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264375] pci_bus 0000:0d: resource 5 [io 0x03e0-0x0cf7] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264377] pci_bus 0000:0d: resource 6 [io 0x03b0-0x03df] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264378] pci_bus 0000:0d: resource 7 [io 0x0d00-0x1fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264379] pci_bus 0000:0d: resource 8 [io 0xe000-0xffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264380] pci_bus 0000:0d: resource 9 [mem 0x000a0000-0x000bffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264382] pci_bus 0000:0d: resource 10 [mem 0xd8000000-0xdfffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264383] pci_bus 0000:0d: resource 11 [mem 0xfed40000-0xfed44fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264384] pci_bus 0000:40: resource 4 [io 0x2000-0x3fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264386] pci_bus 0000:40: resource 5 [io 0x4000-0x5fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264387] pci_bus 0000:40: resource 6 [io 0x6000-0x7fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264388] pci_bus 0000:40: resource 7 [io 0x8000-0x9fff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264389] pci_bus 0000:40: resource 8 [io 0xa000-0xbfff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264391] pci_bus 0000:40: resource 9 [io 0xc000-0xdfff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264392] pci_bus 0000:40: resource 10 [mem 0xd0000000-0xd7ffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264393] pci_bus 0000:41: resource 0 [io 0xd000-0xdfff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264395] pci_bus 0000:41: resource 1 [mem 0xd4000000-0xd4ffffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264396] pci_bus 0000:42: resource 1 [mem 0xd5000000-0xd50fffff] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264397] pci_bus 0000:42: resource 2 [mem 0xd0000000-0xd07fffff 64bit pref] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.264631] NET: Registered protocol family 2 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.270146] IP route cache hash table entries: 524288 (order: 10, 4194304 bytes) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.280256] TCP established hash table entries: 524288 (order: 11, 8388608 bytes) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.290109] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.297912] TCP: Hash tables configured (established 524288 bind 65536) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.305492] TCP reno registered Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.309581] NET: Registered protocol family 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.314661] alloc irq_desc for 23 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.314662] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.314667] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.314672] pci 0000:00:1a.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.333370] pci 0000:00:1a.0: PCI INT A disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.338726] alloc irq_desc for 22 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.338727] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.338730] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.338734] pci 0000:00:1d.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.346452] pci 0000:00:1d.0: PCI INT A disabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.351951] pci 0000:0b:00.0: Boot video device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.352079] Trying to unpack rootfs image as initramfs... Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.524763] Freeing initrd memory: 13548k freed Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.532786] dmar: Unsupported device scope Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.557280] HugeTLB registered 2 MB page size, pre-allocated 0 pages Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.583720] VFS: Disk quotas dquot_6.5.2 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.588332] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.595864] msgmni has been set to 32768 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.601194] alg: No test for stdrng (krng) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.605975] ksign: Installing public key data Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.611033] Loading keyring Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.614341] - Added public key 77A970641DBCAA51 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.619588] - User ID: Red Hat, Inc. (Kernel Module GPG key) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.626110] - Added public key D4A26C9CCD09BEDA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.631352] - User ID: Red Hat Enterprise Linux Driver Update Program Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.641022] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.650137] io scheduler noop registered Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.654704] io scheduler anticipatory registered Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.660043] io scheduler deadline registered Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.665083] io scheduler cfq registered (default) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.670828] pcieport 0000:00:01.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.670858] alloc irq_desc for 95 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.670859] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.670865] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.670873] pcieport 0000:00:01.0: irq 95 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.670958] pcieport 0000:00:01.1: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.670985] alloc irq_desc for 96 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.670986] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.670988] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.670995] pcieport 0000:00:01.1: irq 96 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671071] pcieport 0000:00:02.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671099] alloc irq_desc for 97 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671100] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671103] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671109] pcieport 0000:00:02.0: irq 97 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671184] pcieport 0000:00:02.2: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671210] alloc irq_desc for 98 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671211] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671214] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671221] pcieport 0000:00:02.2: irq 98 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671300] pcieport 0000:00:03.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671326] alloc irq_desc for 99 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671327] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671331] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671337] pcieport 0000:00:03.0: irq 99 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671415] pcieport 0000:00:11.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671464] alloc irq_desc for 100 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671465] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671468] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671475] pcieport 0000:00:11.0: irq 100 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671565] pcieport 0000:00:1c.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671596] alloc irq_desc for 101 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671597] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671601] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671608] pcieport 0000:00:1c.0: irq 101 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671700] pcieport 0000:00:1c.7: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671731] alloc irq_desc for 102 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671732] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671735] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671742] pcieport 0000:00:1c.7: irq 102 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671961] pcieport 0000:40:01.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671990] alloc irq_desc for 103 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671992] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.671997] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.672005] pcieport 0000:40:01.0: irq 103 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.672111] pcieport 0000:40:03.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.672137] alloc irq_desc for 104 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.672138] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.672140] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.672147] pcieport 0000:40:03.0: irq 104 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.672212] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.678648] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.686232] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.693973] intel_idle: MWAIT substates: 0x21120 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.693974] intel_idle: v0.4 model 0x2D Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.693975] intel_idle: lapic_timer_reliable_states 0xffffffff Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.694094] ipmi message handler version 39.2 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.699155] IPMI System Interface driver. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.703832] ipmi_si: Adding SMBIOS-specified kcs state machine Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.710641] ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 10 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.752906] IRQ 10/ipmi_si: IRQF_DISABLED is not guaranteed on shared IRQs Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.760787] Using irq 10 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.762581] ipmi_si: Could not enable interrupts, failed set, using polled mode. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.782250] ipmi: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.791005] IPMI kcs interface initialized Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.795860] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.804410] ACPI: Power Button [PWRF] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.808921] ACPI: acpi_idle yielding to intel_idle Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.816076] ERST: Error Record Serialization Table (ERST) support is initialized. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.824936] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.834564] Non-volatile memory driver v1.3 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.839428] Linux agpgart interface v0.103 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.844284] crash memory driver: version 1.1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 3.849244] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.100240] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.351484] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.360548] 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.367080] 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.374903] brd: module loaded Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.379051] loop: module loaded Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.382773] input: Macintosh mouse button emulation as /devices/virtual/input/input1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.391750] Fixed MDIO Bus: probed Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.395764] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.403284] ehci_hcd 0000:00:1a.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.411478] ehci_hcd 0000:00:1a.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.411480] ehci_hcd 0000:00:1a.0: EHCI Host Controller Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.417548] ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.426128] ehci_hcd 0000:00:1a.0: debug port 2 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.435278] ehci_hcd 0000:00:1a.0: cache line size of 64 is not supported Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.435295] ehci_hcd 0000:00:1a.0: irq 23, io mem 0xde8fd000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.451449] ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.458075] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.465851] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.474200] usb usb1: Product: EHCI Host Controller Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.479841] usb usb1: Manufacturer: Linux 2.6.32-358.23.2.el6.atlas.x86_64 ehci_hcd Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.488685] usb usb1: SerialNumber: 0000:00:1a.0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.494075] usb usb1: configuration #1 chosen from 1 choice Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.500514] hub 1-0:1.0: USB hub found Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.504891] hub 1-0:1.0: 2 ports detected Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.509635] ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.517814] ehci_hcd 0000:00:1d.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.517816] ehci_hcd 0000:00:1d.0: EHCI Host Controller Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.523870] ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.532436] ehci_hcd 0000:00:1d.0: debug port 2 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.541580] ehci_hcd 0000:00:1d.0: cache line size of 64 is not supported Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.541585] Refined TSC clocksource calibration: 2499.999 MHz. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.548298] Switching to clocksource tsc Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.552930] ehci_hcd 0000:00:1d.0: irq 22, io mem 0xde8fe000 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.568556] ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.575179] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.582956] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.591317] usb usb2: Product: EHCI Host Controller Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.596949] usb usb2: Manufacturer: Linux 2.6.32-358.23.2.el6.atlas.x86_64 ehci_hcd Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.605798] usb usb2: SerialNumber: 0000:00:1d.0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.611193] usb usb2: configuration #1 chosen from 1 choice Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.617624] hub 2-0:1.0: USB hub found Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.621998] hub 2-0:1.0: 2 ports detected Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.626713] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.633829] uhci_hcd: USB Universal Host Controller Interface driver Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.641200] PNP: No PS/2 controller found. Probing ports directly. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.649194] mice: PS/2 mouse device common for all mice Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.655317] rtc_cmos 00:04: RTC can wake from S4 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.660690] rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.667700] rtc0: alarms up to one day, y3k, 242 bytes nvram, hpet irqs Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.675971] cpuidle: using governor ladder Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.681847] cpuidle: using governor menu Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.686409] EFI Variables Facility v0.08 2004-May-17 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.692352] usbcore: registered new interface driver hiddev Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.698774] usbcore: registered new interface driver usbhid Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.705187] usbhid: v2.6:USB HID core driver Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.710333] TCP cubic registered Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.714108] Initializing XFRM netlink socket Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.719064] NET: Registered protocol family 17 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.724283] registered taskstats version 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.745617] rtc_cmos 00:04: setting system clock to 2014-07-17 21:43:57 UTC (1405633437) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.754992] Initalizing network drop monitor service Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.760773] Freeing unused kernel memory: 1264k freed Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.766881] Write protecting the kernel read-only data: 10240k Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.774036] Freeing unused kernel memory: 908k freed Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.780176] Freeing unused kernel memory: 1676k freed Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.795391] dracut: dracut-004-303.el6ccs Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.807307] udev: starting version 147 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.812963] usb 1-1: new high speed USB device number 2 using ehci_hcd Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.826817] tg3.c:v3.124 (March 21, 2012) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.831510] alloc irq_desc for 35 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.831512] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.831517] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.831524] tg3 0000:01:00.0: PCI INT A -> GSI 35 (level, low) -> IRQ 35 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.843871] tg3 0000:01:00.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.849471] tg3 0000:01:00.0: eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 90:b1:1c:38:49:d3 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.861636] tg3 0000:01:00.0: eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.873013] tg3 0000:01:00.0: eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.882162] tg3 0000:01:00.0: eth0: dma_rwctrl[00000001] dma_mask[64-bit] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.890023] alloc irq_desc for 38 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.890024] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.890029] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.890034] tg3 0000:01:00.1: PCI INT B -> GSI 38 (level, low) -> IRQ 38 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.897720] tg3 0000:01:00.1: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.903340] tg3 0000:01:00.1: eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 90:b1:1c:38:49:d4 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.915501] tg3 0000:01:00.1: eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.926891] tg3 0000:01:00.1: eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.936036] tg3 0000:01:00.1: eth1: dma_rwctrl[00000001] dma_mask[64-bit] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.943889] alloc irq_desc for 34 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.943890] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.943893] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.943897] tg3 0000:02:00.0: PCI INT A -> GSI 34 (level, low) -> IRQ 34 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.951593] tg3 0000:02:00.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.951855] usb 1-1: New USB device found, idVendor=8087, idProduct=0024 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.959558] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.967785] usb 1-1: configuration #1 chosen from 1 choice Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.968156] tg3 0000:02:00.0: eth2: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 90:b1:1c:38:49:d5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.968159] tg3 0000:02:00.0: eth2: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.968161] tg3 0000:02:00.0: eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.968163] tg3 0000:02:00.0: eth2: dma_rwctrl[00000001] dma_mask[64-bit] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.968233] alloc irq_desc for 36 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.968234] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.968237] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.968242] tg3 0000:02:00.1: PCI INT B -> GSI 36 (level, low) -> IRQ 36 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.968248] tg3 0000:02:00.1: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.983879] tg3 0000:02:00.1: eth3: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 90:b1:1c:38:49:d6 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.983881] tg3 0000:02:00.1: eth3: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.983883] tg3 0000:02:00.1: eth3: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 4.983885] tg3 0000:02:00.1: eth3: dma_rwctrl[00000001] dma_mask[64-bit] Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.046565] mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.046567] mlx4_core: Initializing 0000:42:00.0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.046644] alloc irq_desc for 80 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.046645] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.046650] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.046657] mlx4_core 0000:42:00.0: PCI INT A -> GSI 80 (level, low) -> IRQ 80 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.046690] mlx4_core 0000:42:00.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083768] alloc irq_desc for 105 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083773] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083783] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083803] tg3 0000:01:00.0: irq 105 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083804] alloc irq_desc for 106 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083805] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083808] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083811] tg3 0000:01:00.0: irq 106 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083813] alloc irq_desc for 107 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083814] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083816] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083820] tg3 0000:01:00.0: irq 107 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083821] alloc irq_desc for 108 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083822] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083824] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083827] tg3 0000:01:00.0: irq 108 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083829] alloc irq_desc for 109 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083830] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083832] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083835] tg3 0000:01:00.0: irq 109 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.083909] hub 1-1:1.0: USB hub found Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.088410] hub 1-1:1.0: 6 ports detected Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.196224] usb 2-1: new high speed USB device number 2 using ehci_hcd Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.317760] usb 2-1: New USB device found, idVendor=8087, idProduct=0024 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.325459] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.333687] usb 2-1: configuration #1 chosen from 1 choice Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.340192] hub 2-1:1.0: USB hub found Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.344726] hub 2-1:1.0: 8 ports detected Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.423622] usb 1-1.4: new low speed USB device number 3 using ehci_hcd Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.526209] usb 1-1.4: New USB device found, idVendor=03f0, idProduct=0024 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.534102] usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.542554] usb 1-1.4: Product: HP Basic USB Keyboard Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.548382] usb 1-1.4: Manufacturer: CHICONY Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.553373] usb 1-1.4: configuration #1 chosen from 1 choice Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.565262] input: CHICONY HP Basic USB Keyboard as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.4/1-1.4:1.0/input/input2 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.577965] generic-usb 0003:03F0:0024.0001: input,hidraw0: USB HID v1.10 Keyboard [CHICONY HP Basic USB Keyboard] on usb-0000:00:1a.0-1.4/input0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.665859] usb 1-1.6: new high speed USB device number 4 using ehci_hcd Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.759825] usb 1-1.6: New USB device found, idVendor=0624, idProduct=0248 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.767718] usb 1-1.6: New USB device strings: Mfr=1, Product=2, SerialNumber=3 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.776177] usb 1-1.6: Product: Gadget USB HUB Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.781325] usb 1-1.6: Manufacturer: no manufacturer Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.787056] usb 1-1.6: SerialNumber: 0123456789 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.792343] usb 1-1.6: configuration #1 chosen from 1 choice Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.799135] hub 1-1.6:1.0: USB hub found Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 5.803804] hub 1-1.6:1.0: 6 ports detected Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 6.092268] usb 1-1.6.1: new high speed USB device number 5 using ehci_hcd Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 6.186336] usb 1-1.6.1: New USB device found, idVendor=0624, idProduct=0249 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 6.194434] usb 1-1.6.1: New USB device strings: Mfr=4, Product=5, SerialNumber=6 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 6.203103] usb 1-1.6.1: Product: Keyboard/Mouse Function Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 6.209353] usb 1-1.6.1: Manufacturer: Avocent Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 6.214510] usb 1-1.6.1: SerialNumber: 20120917 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 6.219795] usb 1-1.6.1: configuration #1 chosen from 1 choice Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 6.232078] input: Avocent Keyboard/Mouse Function as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1-1.6.1:1.0/input/input3 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 6.245943] generic-usb 0003:0624:0249.0002: input,hidraw1: USB HID v1.00 Keyboard [Avocent Keyboard/Mouse Function] on usb-0000:00:1a.0-1.6.1/input0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 6.261822] input: Avocent Keyboard/Mouse Function as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1-1.6.1:1.1/input/input4 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 6.275672] generic-usb 0003:0624:0249.0003: input,hidraw2: USB HID v1.00 Mouse [Avocent Keyboard/Mouse Function] on usb-0000:00:1a.0-1.6.1/input1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 6.291224] input: Avocent Keyboard/Mouse Function as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1-1.6.1:1.2/input/input5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 6.305076] generic-usb 0003:0624:0249.0004: input,hidraw3: USB HID v1.00 Mouse [Avocent Keyboard/Mouse Function] on usb-0000:00:1a.0-1.6.1/input2 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 8.759700] tg3 0000:01:00.0: eth0: Link is up at 1000 Mbps, full duplex Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 8.767378] tg3 0000:01:00.0: eth0: Flow control is on for TX and on for RX Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 8.775345] tg3 0000:01:00.0: eth0: EEE is enabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 11.623685] mlx4_core 0000:42:00.0: 64B EQEs/CQEs supported by the device but not enabled Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371748] alloc irq_desc for 110 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371749] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371754] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371759] mlx4_core 0000:42:00.0: irq 110 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371760] alloc irq_desc for 111 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371761] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371763] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371766] mlx4_core 0000:42:00.0: irq 111 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371768] alloc irq_desc for 112 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371769] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371770] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371774] mlx4_core 0000:42:00.0: irq 112 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371775] alloc irq_desc for 113 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371776] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371778] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371781] mlx4_core 0000:42:00.0: irq 113 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371783] alloc irq_desc for 114 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371784] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371786] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371789] mlx4_core 0000:42:00.0: irq 114 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371790] alloc irq_desc for 115 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371791] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371793] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371796] mlx4_core 0000:42:00.0: irq 115 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371797] alloc irq_desc for 116 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371798] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371800] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371803] mlx4_core 0000:42:00.0: irq 116 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371804] alloc irq_desc for 117 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371805] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371807] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371811] mlx4_core 0000:42:00.0: irq 117 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371812] alloc irq_desc for 118 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371812] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371815] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371818] mlx4_core 0000:42:00.0: irq 118 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371819] alloc irq_desc for 119 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371820] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371822] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371825] mlx4_core 0000:42:00.0: irq 119 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371826] alloc irq_desc for 120 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371827] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371829] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371832] mlx4_core 0000:42:00.0: irq 120 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371833] alloc irq_desc for 121 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371834] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371836] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371839] mlx4_core 0000:42:00.0: irq 121 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371840] alloc irq_desc for 122 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371841] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371843] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.371846] mlx4_core 0000:42:00.0: irq 122 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.422212] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.0 (Dec 2011) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.482442] RPC: Registered named UNIX socket transport module. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.489278] RPC: Registered udp transport module. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.494719] RPC: Registered tcp transport module. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.500160] RPC: Registered tcp NFSv4.1 backchannel transport module. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.519792] Slow work thread pool: Starting up Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.524981] Slow work thread pool: Ready Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.529623] FS-Cache: Loaded Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.558829] Registering the id_resolver key type Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.564201] FS-Cache: Netfs 'nfs' registered for caching Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.620770] NET: Registered protocol family 10 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.626671] lo: Disabled Privacy Extensions Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.646707] dracut: Mounted root filesystem 172.30.16.12:/images/nfsroot/hosts/atlas-mds3 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 12.770748] dracut: Switching root Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 23.151698] eth0: no IPv6 routers present Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.303241] udev: starting version 147 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.375954] iTCO_vendor_support: vendor-support=0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.391662] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.398413] iTCO_wdt: Found a Patsburg TCO device (Version=2, TCOBASE=0x0860) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.406635] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.424691] ahci 0000:00:1f.2: version 3.0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.424706] alloc irq_desc for 20 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.424708] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.424730] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.424737] ahci 0000:00:1f.2: PCI INT A -> GSI 20 (level, low) -> IRQ 20 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.432663] alloc irq_desc for 123 on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.432665] alloc kstat_irqs on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.432671] alloc irq_2_iommu on node 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.432680] ahci 0000:00:1f.2: irq 123 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.443399] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 3 Gbps 0x3f impl SATA mode Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.453095] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.462378] ahci 0000:00:1f.2: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.473584] scsi0 : ahci Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.476814] scsi1 : ahci Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.479955] scsi2 : ahci Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.483078] scsi3 : ahci Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.486204] scsi4 : ahci Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.489327] scsi5 : ahci Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.492434] ata1: SATA max UDMA/133 abar m2048@0xde8ff000 port 0xde8ff100 irq 123 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.501144] ata2: SATA max UDMA/133 abar m2048@0xde8ff000 port 0xde8ff180 irq 123 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.509927] ata3: SATA max UDMA/133 abar m2048@0xde8ff000 port 0xde8ff200 irq 123 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.518640] ata4: SATA max UDMA/133 abar m2048@0xde8ff000 port 0xde8ff280 irq 123 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.527340] ata5: SATA max UDMA/133 abar m2048@0xde8ff000 port 0xde8ff300 irq 123 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.536042] ata6: SATA max UDMA/133 abar m2048@0xde8ff000 port 0xde8ff380 irq 123 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.849941] ata2: SATA link down (SStatus 0 SControl 300) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.851877] ata4: SATA link down (SStatus 0 SControl 300) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.851906] ata5: SATA link down (SStatus 0 SControl 300) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.851912] ata6: SATA link down (SStatus 0 SControl 300) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.851932] ata3: SATA link down (SStatus 0 SControl 300) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.851935] ata1: SATA link down (SStatus 0 SControl 300) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.935380] EDAC MC: Ver: 2.1.0 Mar 28 2014 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.945521] EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.952433] EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.959540] EDAC sbridge: Seeking for: dev 0e.0 PCI ID 8086:3ca0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.966447] EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.973371] EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.980278] EDAC sbridge: Seeking for: dev 0f.0 PCI ID 8086:3ca8 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.987185] EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 47.994102] EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.001004] EDAC sbridge: Seeking for: dev 0f.1 PCI ID 8086:3c71 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.007905] EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.014817] EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.021727] EDAC sbridge: Seeking for: dev 0f.2 PCI ID 8086:3caa Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.028632] EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.035577] EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.042484] EDAC sbridge: Seeking for: dev 0f.3 PCI ID 8086:3cab Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.049383] EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.056286] EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.063206] EDAC sbridge: Seeking for: dev 0f.4 PCI ID 8086:3cac Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.070110] EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.077030] EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.083939] EDAC sbridge: Seeking for: dev 0f.5 PCI ID 8086:3cad Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.090844] EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.097766] EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.104674] EDAC sbridge: Seeking for: dev 11.0 PCI ID 8086:3cb8 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.111577] EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.118494] EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.125406] EDAC sbridge: Seeking for: dev 0c.6 PCI ID 8086:3cf4 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.132313] EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.139226] EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.146135] EDAC sbridge: Seeking for: dev 0c.7 PCI ID 8086:3cf6 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.153038] EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.159960] EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.166887] EDAC sbridge: Seeking for: dev 0d.6 PCI ID 8086:3cf5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.174065] EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:3f:0e.0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.184925] EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:7f:0e.0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.195512] EDAC sbridge: Driver loaded. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.318064] dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.333877] microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.340701] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.367641] microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.374451] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.387516] microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.394318] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.406648] microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.413459] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.426355] microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.433156] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.445953] microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.452763] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.465662] microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.472457] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.484997] microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.491800] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.504737] microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.511538] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.524154] microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.530967] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.543854] microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.550751] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.563232] microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.570143] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.582667] microcode: CPU12 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.589560] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.602098] microcode: CPU13 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.609092] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.621876] microcode: CPU14 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.628778] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.641267] microcode: CPU15 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.648271] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.660909] microcode: CPU16 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.667808] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.680515] microcode: CPU17 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.687420] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.699935] microcode: CPU18 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.706835] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.719559] microcode: CPU19 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.726465] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.738964] microcode: CPU20 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.745865] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.758844] microcode: CPU21 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.765747] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.778437] microcode: CPU22 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.785605] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.786094] udev: renamed network interface eth3 to em4 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.804365] microcode: CPU23 sig=0x206d7, pf=0x1, revision=0x710 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.811271] platform microcode: firmware: requesting intel-ucode/06-2d-07 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.823734] Microcode Update Driver: v2.00 , Peter Oruba Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.855081] mpt2sas version 13.101.00.00 loaded Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.860532] mpt2sas0: The max_sectors value is set to 4096 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.866855] scsi6 : Fusion MPT SAS Host Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.871566] alloc irq_desc for 66 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.871568] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.871574] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.871582] mpt2sas 0000:41:00.0: PCI INT A -> GSI 66 (level, low) -> IRQ 66 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.879687] mpt2sas 0000:41:00.0: setting latency timer to 64 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.879709] mpt2sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (264436592 kB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.889151] alloc irq_desc for 124 on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.889152] alloc kstat_irqs on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.889155] alloc irq_2_iommu on node 1 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.889160] mpt2sas 0000:41:00.0: irq 124 for MSI/MSI-X Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.889205] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 124 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.895136] mpt2sas0: iomem(0x00000000d4ff0000), mapped(0xffffc90029a40000), size(65536) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 48.904471] mpt2sas0: ioport(0x000000000000dc00), size(256) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 49.026192] mpt2sas0: Allocated physical memory: size(6173 kB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 49.032899] mpt2sas0: Current Controller Queue Depth(2746), Max Controller Queue Depth(2879) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 49.042622] mpt2sas0: Scatter Gather Elements per IO(128) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 49.107373] mpt2sas0: LSISAS2008: FWVersion(07.15.08.00), ChipRevision(0x03), BiosVersion(07.02.03.00) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 49.118068] mpt2sas0: Dell 6Gbps SAS HBA: Vendor(0x1000), Device(0x0072), SSVID(0x1028), SSDID(0x1F1C) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 49.128762] mpt2sas0: Protocol=(Initiator,Target), Capabilities=(Raid,TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 49.143448] mpt2sas0: sending port enable !! Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 49.724880] udev: renamed network interface eth1 to em2 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 50.657361] udev: renamed network interface eth2 to em3 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 50.747970] mpt2sas0: host_add: handle(0x0001), sas_addr(0x590b11c013962a00), phys(8) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 50.758335] mpt2sas0: expander_add: handle(0x0009), parent(0x0001), sas_addr(0x500062b1000629ff), phys(66) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 51.003175] mpt2sas0: expander_add: handle(0x0016), parent(0x0002), sas_addr(0x500062b10006937f), phys(66) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.260998] mpt2sas0: port enable: SUCCESS Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.268013] scsi 6:0:0:0: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.277351] scsi 6:0:0:0: SSP: handle(0x000b), sas_addr(0x50080e529398c044), phy(20), device_name(0x520e085044c09893) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.289495] scsi 6:0:0:0: SSP: enclosure_logical_id(0x500062b1000629ff), slot(20) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.298149] scsi 6:0:0:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.309848] scsi 6:0:0:1: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.319198] scsi 6:0:0:1: SSP: handle(0x000b), sas_addr(0x50080e529398c044), phy(20), device_name(0x520e085044c09893) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.331349] scsi 6:0:0:1: SSP: enclosure_logical_id(0x500062b1000629ff), slot(20) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.340002] scsi 6:0:0:1: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.350780] scsi 6:0:0:2: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.360110] scsi 6:0:0:2: SSP: handle(0x000b), sas_addr(0x50080e529398c044), phy(20), device_name(0x520e085044c09893) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.372265] scsi 6:0:0:2: SSP: enclosure_logical_id(0x500062b1000629ff), slot(20) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.380920] scsi 6:0:0:2: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.391706] scsi 6:0:0:3: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.401047] scsi 6:0:0:3: SSP: handle(0x000b), sas_addr(0x50080e529398c044), phy(20), device_name(0x520e085044c09893) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.413205] scsi 6:0:0:3: SSP: enclosure_logical_id(0x500062b1000629ff), slot(20) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.421859] scsi 6:0:0:3: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.434242] scsi 6:0:1:0: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.443589] scsi 6:0:1:0: SSP: handle(0x000c), sas_addr(0x50080e529398c040), phy(24), device_name(0x0000000000000000) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.455750] scsi 6:0:1:0: SSP: enclosure_logical_id(0x500062b1000629ff), slot(24) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.464405] scsi 6:0:1:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.475926] scsi 6:0:1:1: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.485269] scsi 6:0:1:1: SSP: handle(0x000c), sas_addr(0x50080e529398c040), phy(24), device_name(0x0000000000000000) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.497423] scsi 6:0:1:1: SSP: enclosure_logical_id(0x500062b1000629ff), slot(24) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.506082] scsi 6:0:1:1: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.516854] scsi 6:0:1:2: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.526197] scsi 6:0:1:2: SSP: handle(0x000c), sas_addr(0x50080e529398c040), phy(24), device_name(0x0000000000000000) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.538350] scsi 6:0:1:2: SSP: enclosure_logical_id(0x500062b1000629ff), slot(24) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.547003] scsi 6:0:1:2: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.557794] scsi 6:0:1:3: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.567137] scsi 6:0:1:3: SSP: handle(0x000c), sas_addr(0x50080e529398c040), phy(24), device_name(0x0000000000000000) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.579282] scsi 6:0:1:3: SSP: enclosure_logical_id(0x500062b1000629ff), slot(24) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.587925] scsi 6:0:1:3: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.600179] scsi 6:0:2:0: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.609541] scsi 6:0:2:0: SSP: handle(0x000d), sas_addr(0x50080e5294180044), phy(28), device_name(0x520e085044001894) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.621695] scsi 6:0:2:0: SSP: enclosure_logical_id(0x500062b1000629ff), slot(28) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.630339] scsi 6:0:2:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.641783] scsi 6:0:2:1: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.651126] scsi 6:0:2:1: SSP: handle(0x000d), sas_addr(0x50080e5294180044), phy(28), device_name(0x520e085044001894) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.663279] scsi 6:0:2:1: SSP: enclosure_logical_id(0x500062b1000629ff), slot(28) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.671933] scsi 6:0:2:1: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.682551] scsi 6:0:2:2: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.691889] scsi 6:0:2:2: SSP: handle(0x000d), sas_addr(0x50080e5294180044), phy(28), device_name(0x520e085044001894) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.704042] scsi 6:0:2:2: SSP: enclosure_logical_id(0x500062b1000629ff), slot(28) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.712694] scsi 6:0:2:2: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.723230] scsi 6:0:2:3: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.732565] scsi 6:0:2:3: SSP: handle(0x000d), sas_addr(0x50080e5294180044), phy(28), device_name(0x520e085044001894) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.744720] scsi 6:0:2:3: SSP: enclosure_logical_id(0x500062b1000629ff), slot(28) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.753378] scsi 6:0:2:3: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.765192] scsi 6:0:3:0: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.774556] scsi 6:0:3:0: SSP: handle(0x000e), sas_addr(0x50080e5294180040), phy(36), device_name(0x520e085040001894) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.786707] scsi 6:0:3:0: SSP: enclosure_logical_id(0x500062b1000629ff), slot(36) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.795360] scsi 6:0:3:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.806461] scsi 6:0:3:1: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.815887] scsi 6:0:3:1: SSP: handle(0x000e), sas_addr(0x50080e5294180040), phy(36), device_name(0x520e085040001894) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.828041] scsi 6:0:3:1: SSP: enclosure_logical_id(0x500062b1000629ff), slot(36) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.836696] scsi 6:0:3:1: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.847277] scsi 6:0:3:2: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.856617] scsi 6:0:3:2: SSP: handle(0x000e), sas_addr(0x50080e5294180040), phy(36), device_name(0x520e085040001894) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.868778] scsi 6:0:3:2: SSP: enclosure_logical_id(0x500062b1000629ff), slot(36) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.877436] scsi 6:0:3:2: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.888224] scsi 6:0:3:3: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.897555] scsi 6:0:3:3: SSP: handle(0x000e), sas_addr(0x50080e5294180040), phy(36), device_name(0x520e085040001894) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.909706] scsi 6:0:3:3: SSP: enclosure_logical_id(0x500062b1000629ff), slot(36) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.918361] scsi 6:0:3:3: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.929723] scsi 6:0:4:0: Enclosure LSI SAS616x 0e05 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.939078] scsi 6:0:4:0: SSP: handle(0x0015), sas_addr(0x500062b1000629fd), phy(64), device_name(0x0000000000000000) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.951231] scsi 6:0:4:0: SSP: enclosure_logical_id(0x500062b1000629ff), slot(64) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.959885] scsi 6:0:4:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.977498] scsi 6:0:5:0: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.986925] scsi 6:0:5:0: SSP: handle(0x0017), sas_addr(0x50080e529398c034), phy(0), device_name(0x0000000000000000) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 56.998983] scsi 6:0:5:0: SSP: enclosure_logical_id(0x500062b10006937f), slot(0) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.007540] scsi 6:0:5:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.018706] scsi 6:0:5:1: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.028043] scsi 6:0:5:1: SSP: handle(0x0017), sas_addr(0x50080e529398c034), phy(0), device_name(0x0000000000000000) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.040105] scsi 6:0:5:1: SSP: enclosure_logical_id(0x500062b10006937f), slot(0) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.048663] scsi 6:0:5:1: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.059284] scsi 6:0:5:2: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.068616] scsi 6:0:5:2: SSP: handle(0x0017), sas_addr(0x50080e529398c034), phy(0), device_name(0x0000000000000000) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.080671] scsi 6:0:5:2: SSP: enclosure_logical_id(0x500062b10006937f), slot(0) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.089226] scsi 6:0:5:2: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.099759] scsi 6:0:5:3: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.109097] scsi 6:0:5:3: SSP: handle(0x0017), sas_addr(0x50080e529398c034), phy(0), device_name(0x0000000000000000) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.121154] scsi 6:0:5:3: SSP: enclosure_logical_id(0x500062b10006937f), slot(0) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.129704] scsi 6:0:5:3: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.141825] scsi 6:0:6:0: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.151189] scsi 6:0:6:0: SSP: handle(0x0018), sas_addr(0x50080e529398c030), phy(4), device_name(0x520e085030c09893) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.163242] scsi 6:0:6:0: SSP: enclosure_logical_id(0x500062b10006937f), slot(4) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.171789] scsi 6:0:6:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.182979] scsi 6:0:6:1: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.192313] scsi 6:0:6:1: SSP: handle(0x0018), sas_addr(0x50080e529398c030), phy(4), device_name(0x520e085030c09893) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.206324] scsi 6:0:6:1: SSP: enclosure_logical_id(0x500062b10006937f), slot(4) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.214870] scsi 6:0:6:1: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.225409] scsi 6:0:6:2: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.234746] scsi 6:0:6:2: SSP: handle(0x0018), sas_addr(0x50080e529398c030), phy(4), device_name(0x520e085030c09893) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.246794] scsi 6:0:6:2: SSP: enclosure_logical_id(0x500062b10006937f), slot(4) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.255349] scsi 6:0:6:2: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.265892] scsi 6:0:6:3: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.275227] scsi 6:0:6:3: SSP: handle(0x0018), sas_addr(0x50080e529398c030), phy(4), device_name(0x520e085030c09893) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.287283] scsi 6:0:6:3: SSP: enclosure_logical_id(0x500062b10006937f), slot(4) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.295838] scsi 6:0:6:3: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.307678] scsi 6:0:7:0: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.317023] scsi 6:0:7:0: SSP: handle(0x0019), sas_addr(0x50080e5294180030), phy(24), device_name(0x0000000000000000) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.329183] scsi 6:0:7:0: SSP: enclosure_logical_id(0x500062b10006937f), slot(24) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.337837] scsi 6:0:7:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.348914] scsi 6:0:7:1: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.358243] scsi 6:0:7:1: SSP: handle(0x0019), sas_addr(0x50080e5294180030), phy(24), device_name(0x0000000000000000) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.370394] scsi 6:0:7:1: SSP: enclosure_logical_id(0x500062b10006937f), slot(24) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.379054] scsi 6:0:7:1: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.389590] scsi 6:0:7:2: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.398925] scsi 6:0:7:2: SSP: handle(0x0019), sas_addr(0x50080e5294180030), phy(24), device_name(0x0000000000000000) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.411084] scsi 6:0:7:2: SSP: enclosure_logical_id(0x500062b10006937f), slot(24) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.419736] scsi 6:0:7:2: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.430267] scsi 6:0:7:3: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.439599] scsi 6:0:7:3: SSP: handle(0x0019), sas_addr(0x50080e5294180030), phy(24), device_name(0x0000000000000000) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.451753] scsi 6:0:7:3: SSP: enclosure_logical_id(0x500062b10006937f), slot(24) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.460404] scsi 6:0:7:3: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.472252] scsi 6:0:8:0: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.481589] scsi 6:0:8:0: SSP: handle(0x001a), sas_addr(0x50080e5294180034), phy(28), device_name(0x520e085034001894) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.493742] scsi 6:0:8:0: SSP: enclosure_logical_id(0x500062b10006937f), slot(28) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.502393] scsi 6:0:8:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.513459] scsi 6:0:8:1: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.522799] scsi 6:0:8:1: SSP: handle(0x001a), sas_addr(0x50080e5294180034), phy(28), device_name(0x520e085034001894) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.534953] scsi 6:0:8:1: SSP: enclosure_logical_id(0x500062b10006937f), slot(28) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.543606] scsi 6:0:8:1: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.554269] scsi 6:0:8:2: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.563605] scsi 6:0:8:2: SSP: handle(0x001a), sas_addr(0x50080e5294180034), phy(28), device_name(0x520e085034001894) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.575758] scsi 6:0:8:2: SSP: enclosure_logical_id(0x500062b10006937f), slot(28) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.584403] scsi 6:0:8:2: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.594936] scsi 6:0:8:3: Direct-Access NETAPP INF-01-00 0810 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.604271] scsi 6:0:8:3: SSP: handle(0x001a), sas_addr(0x50080e5294180034), phy(28), device_name(0x520e085034001894) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.616415] scsi 6:0:8:3: SSP: enclosure_logical_id(0x500062b10006937f), slot(28) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.625071] scsi 6:0:8:3: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.636055] scsi 6:0:9:0: Enclosure LSI SAS616x 0e05 PQ: 0 ANSI: 5 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.645406] scsi 6:0:9:0: SSP: handle(0x0022), sas_addr(0x500062b10006937d), phy(64), device_name(0x0000000000000000) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.657558] scsi 6:0:9:0: SSP: enclosure_logical_id(0x500062b10006937f), slot(64) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.666210] scsi 6:0:9:0: qdepth(254), tagged(1), simple(1), ordered(0), scsi_level(6), cmd_que(1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.695058] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.720365] ACPI Error: No handler for Region [SYSI] (ffff882053403420) [IPMI] (20090903/evregion-319) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.731260] ACPI Error: Region IPMI(7) has no handler (20090903/exfldio-295) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.739511] ACPI Error (psparse-0537): Method parse/execution failed [\_SB_.PMI0._GHL] (Node ffff882053400790), AE_NOT_EXIST Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.752620] ACPI Error (psparse-0537): Method parse/execution failed [\_SB_.PMI0._PMC] (Node ffff8820534007e0), AE_NOT_EXIST Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.765729] ACPI Exception: AE_NOT_EXIST, Evaluating _PMC (20090903/power_meter-759) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.780891] ACPI: WMI: Mapper loaded Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.841988] sd 6:0:0:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.860742] sd 6:0:0:1: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.879404] sd 6:0:0:2: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.898101] sd 6:0:0:3: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.916785] sd 6:0:1:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.935465] sd 6:0:1:1: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.954139] sd 6:0:1:2: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.972833] sd 6:0:1:3: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 57.991541] sd 6:0:5:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.010244] sd 6:0:5:1: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.028935] sd 6:0:5:2: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.047628] sd 6:0:5:3: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.066318] sd 6:0:6:0: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.085004] sd 6:0:6:2: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.103695] sd 6:0:6:1: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.122376] sd 6:0:6:3: Warning! Received an indication that the operating parameters on this target have changed. The Linux SCSI layer does not automatically adjust these parameters. Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141152] sd 6:0:2:1: [sdj] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141155] sd 6:0:3:2: [sdo] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141169] sd 6:0:3:0: [sdm] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141174] sd 6:0:2:2: [sdk] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141179] sd 6:0:2:0: [sdi] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141187] sd 6:0:3:1: [sdn] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141202] sd 6:0:2:3: [sdl] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141216] sd 6:0:7:1: [sdz] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141222] sd 6:0:3:3: [sdp] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141236] sd 6:0:7:0: [sdy] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141247] sd 6:0:7:2: [sdaa] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141286] sd 6:0:8:0: [sdac] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141289] sd 6:0:0:0: [sda] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141293] sd 6:0:8:3: [sdaf] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141296] sd 6:0:0:2: [sdc] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141302] sd 6:0:7:3: [sdab] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141306] sd 6:0:1:0: [sde] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141316] sd 6:0:1:2: [sdg] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141321] sd 6:0:1:3: [sdh] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141328] sd 6:0:8:1: [sdad] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141339] sd 6:0:8:2: [sdae] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141358] sd 6:0:0:1: [sdb] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141361] sd 6:0:5:2: [sds] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141367] sd 6:0:5:3: [sdt] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141383] sd 6:0:5:0: [sdq] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141387] sd 6:0:6:0: [sdu] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141420] sd 6:0:5:1: [sdr] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141423] sd 6:0:0:3: [sdd] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141426] sd 6:0:6:2: [sdw] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141428] sd 6:0:6:1: [sdv] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141432] sd 6:0:6:3: [sdx] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.141445] sd 6:0:1:1: [sdf] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142035] sd 6:0:2:2: [sdk] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142037] sd 6:0:2:2: [sdk] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142050] sd 6:0:3:0: [sdm] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142052] sd 6:0:3:0: [sdm] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142059] sd 6:0:2:0: [sdi] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142067] sd 6:0:2:0: [sdi] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142098] sd 6:0:7:1: [sdz] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142099] sd 6:0:7:1: [sdz] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142113] sd 6:0:3:1: [sdn] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142115] sd 6:0:3:1: [sdn] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142121] sd 6:0:7:2: [sdaa] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142123] sd 6:0:7:2: [sdaa] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142127] sd 6:0:8:3: [sdaf] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142128] sd 6:0:8:3: [sdaf] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142162] sd 6:0:3:3: [sdp] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142164] sd 6:0:3:3: [sdp] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142167] sd 6:0:8:0: [sdac] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142169] sd 6:0:8:0: [sdac] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142212] sd 6:0:1:0: [sde] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142213] sd 6:0:1:0: [sde] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142225] sd 6:0:0:0: [sda] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142227] sd 6:0:3:2: [sdo] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142230] sd 6:0:0:0: [sda] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142231] sd 6:0:3:2: [sdo] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142237] sd 6:0:1:2: [sdg] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142238] sd 6:0:1:2: [sdg] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142259] sd 6:0:0:2: [sdc] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142261] sd 6:0:0:2: [sdc] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142263] sd 6:0:5:2: [sds] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142265] sd 6:0:5:2: [sds] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142271] sd 6:0:7:3: [sdab] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142273] sd 6:0:2:3: [sdl] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142275] sd 6:0:7:3: [sdab] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142277] sd 6:0:2:3: [sdl] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142279] sd 6:0:5:3: [sdt] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142281] sd 6:0:5:3: [sdt] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142295] sd 6:0:5:0: [sdq] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142296] sd 6:0:5:0: [sdq] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142298] sd 6:0:6:2: [sdw] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142299] sd 6:0:6:2: [sdw] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142322] sd 6:0:6:0: [sdu] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142324] sd 6:0:6:0: [sdu] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142331] sd 6:0:6:1: [sdv] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142332] sd 6:0:6:1: [sdv] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142348] sd 6:0:2:2: [sdk] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142351] sd 6:0:0:1: [sdb] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142353] sd 6:0:0:1: [sdb] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142365] sd 6:0:5:1: [sdr] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142369] sd 6:0:3:0: [sdm] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142382] sd 6:0:5:1: [sdr] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142397] sd 6:0:2:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142400] sd 6:0:7:0: [sdy] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142404] sd 6:0:7:1: [sdz] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142407] sd 6:0:7:0: [sdy] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142410] sd 6:0:8:2: [sdae] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142412] sd 6:0:8:1: [sdad] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142415] sd 6:0:8:2: [sdae] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142417] sd 6:0:8:1: [sdad] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142418] sd 6:0:6:3: [sdx] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142421] sd 6:0:6:3: [sdx] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142442] sd 6:0:1:1: [sdf] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142444] sd 6:0:1:1: [sdf] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142452] sd 6:0:7:2: [sdaa] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142456] sd 6:0:3:1: [sdn] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142463] sd 6:0:3:3: [sdp] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142481] sd 6:0:8:0: [sdac] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142494] sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142501] sd 6:0:0:3: [sdd] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142504] sd 6:0:0:3: [sdd] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142511] sd 6:0:1:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142527] sd 6:0:8:3: [sdaf] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142533] sd 6:0:1:2: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142548] sd 6:0:3:2: [sdo] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142555] sd 6:0:2:3: [sdl] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142558] sd 6:0:7:3: [sdab] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142561] sd 6:0:0:2: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142576] sd 6:0:5:3: [sdt] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142598] sd 6:0:5:0: [sdq] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142606] sd 6:0:6:1: [sdv] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142624] sd 6:0:8:1: [sdad] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142634] sd 6:0:8:2: [sdae] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142650] sd 6:0:1:1: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142655] sd 6:0:5:2: [sds] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142658] sd 6:0:5:1: [sdr] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142661] sd 6:0:1:3: [sdh] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142663] sd 6:0:0:3: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142666] sd 6:0:1:3: [sdh] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142669] sd 6:0:0:1: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142800] sd 6:0:1:3: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.142836] sd 6:0:6:3: [sdx] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.143005] sd 6:0:6:0: [sdu] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.143010] sd 6:0:6:2: [sdw] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.143015] sd 6:0:7:0: [sdy] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.143037] sdm: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.143298] sdk: unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.911579] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.911861] sd 6:0:2:1: [sdj] Write Protect is off Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.911863] sd 6:0:2:1: [sdj] Mode Sense: 83 00 10 08 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.911944] sd 6:0:2:1: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.931894] sdn: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.931994] sd 6:0:3:0: [sdm] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.939688] sdz: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.940591] sd 6:0:2:2: [sdk] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.947532] sdd: unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.952379] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.956625] sdc: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.958676] sdg: unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.963884] sda: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.964314] sd 6:0:0:2: [sdc] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.967904] sd 6:0:3:1: [sdn] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.971986] sd 6:0:7:1: [sdz] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.982363] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.986549] sdad: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.988720] sdo: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.989143] sd 6:0:1:2: [sdg] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 58.996891] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.001087] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.001635] sd 6:0:8:1: [sdad] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.010818] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.011218] sd 6:0:3:2: [sdo] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.020374] sdv: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.020785] sd 6:0:0:0: [sda] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.028152] sdae: unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.033023] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.037246] sdi: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.039327] sdt: unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.044526] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.048769] sdj: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.048807] sd 6:0:8:2: [sdae] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.048854] sd 6:0:6:1: [sdv] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.049156] sd 6:0:2:0: [sdi] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.049162] sd 6:0:5:3: [sdt] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.072707] sdh: unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.077517] sdab: unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.080092] sd 6:0:2:1: [sdj] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.087770] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.088172] sd 6:0:0:3: [sdd] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.097322] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.097748] sd 6:0:1:3: [sdh] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.106895] sdx: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.107287] sd 6:0:7:3: [sdab] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.115052] sdl: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.117123] sdac: unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.122425] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.122832] sd 6:0:2:3: [sdl] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.134062] sds: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.134440] sd 6:0:8:0: [sdac] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.141945] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.146135] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.150393] sdaa: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.150824] sd 6:0:6:3: [sdx] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.150865] sd 6:0:5:2: [sds] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.163709] sdp: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.165773] sdr: unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.171024] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.171447] sd 6:0:3:3: [sdp] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.180621] sde: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.181026] sd 6:0:5:1: [sdr] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.188419] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.192657] sdaf: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.193439] sd 6:0:7:2: [sdaa] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.200674] sdw: unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.205504] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.209731] sdb: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.210198] sd 6:0:6:2: [sdw] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.210251] sd 6:0:8:3: [sdaf] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.222964] sdu: unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.227750] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.232045] sdf: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.232475] sd 6:0:6:0: [sdu] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.239828] sdy: unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.244665] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.248941] sdq: Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.249063] sd 6:0:0:1: [sdb] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.256797] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.257146] sd 6:0:1:0: [sde] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.257159] sd 6:0:1:1: [sdf] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.271671] unknown partition table Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.276317] sd 6:0:7:0: [sdy] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.281671] sd 6:0:5:0: [sdq] Attached SCSI disk Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.332700] sd 6:0:0:0: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.338471] sd 6:0:0:0: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.346257] sd 6:0:0:0: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.351996] sd 6:0:0:1: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.357707] sd 6:0:0:1: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.365498] sd 6:0:0:1: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.371238] sd 6:0:0:2: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.376949] sd 6:0:0:2: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.384731] sd 6:0:0:2: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.390469] sd 6:0:0:3: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.396167] sd 6:0:0:3: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.403963] sd 6:0:0:3: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.409698] sd 6:0:1:0: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.415398] sd 6:0:1:0: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.423188] sd 6:0:1:0: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.428924] sd 6:0:1:1: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.434620] sd 6:0:1:1: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.442413] sd 6:0:1:1: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.448147] sd 6:0:1:2: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.453856] sd 6:0:1:2: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.461657] sd 6:0:1:2: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.467394] sd 6:0:1:3: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.473104] sd 6:0:1:3: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.480900] sd 6:0:1:3: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.486638] sd 6:0:2:0: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.492369] sd 6:0:2:0: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.500169] sd 6:0:2:0: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.505903] sd 6:0:2:1: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.511557] sd 6:0:2:1: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.519333] sd 6:0:2:1: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.525070] sd 6:0:2:2: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.530707] sd 6:0:2:2: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.538490] sd 6:0:2:2: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.544226] sd 6:0:2:3: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.549869] sd 6:0:2:3: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.557646] sd 6:0:2:3: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.563367] sd 6:0:3:0: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.569011] sd 6:0:3:0: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.576786] sd 6:0:3:0: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.582525] sd 6:0:3:1: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.588174] sd 6:0:3:1: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.595950] sd 6:0:3:1: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.601686] sd 6:0:3:2: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.607332] sd 6:0:3:2: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.615105] sd 6:0:3:2: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.620846] sd 6:0:3:3: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.626492] sd 6:0:3:3: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.634270] sd 6:0:3:3: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.646885] sd 6:0:5:0: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.652537] sd 6:0:5:0: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.660317] sd 6:0:5:0: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.666053] sd 6:0:5:1: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.671701] sd 6:0:5:1: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.679473] sd 6:0:5:1: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.685210] sd 6:0:5:2: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.690857] sd 6:0:5:2: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.698636] sd 6:0:5:2: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.704375] sd 6:0:5:3: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.710020] sd 6:0:5:3: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.717797] sd 6:0:5:3: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.723533] sd 6:0:6:0: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.729180] sd 6:0:6:0: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.736959] sd 6:0:6:0: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.742699] sd 6:0:6:1: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.748335] sd 6:0:6:1: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.756112] sd 6:0:6:1: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.761842] sd 6:0:6:2: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.767484] sd 6:0:6:2: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.775261] sd 6:0:6:2: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.780999] sd 6:0:6:3: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.786630] sd 6:0:6:3: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.794401] sd 6:0:6:3: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.800137] sd 6:0:7:0: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.805781] sd 6:0:7:0: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.813557] sd 6:0:7:0: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.819287] sd 6:0:7:1: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.824940] sd 6:0:7:1: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.832720] sd 6:0:7:1: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.838455] sd 6:0:7:2: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.844097] sd 6:0:7:2: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.851866] sd 6:0:7:2: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.857602] sd 6:0:7:3: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.863238] sd 6:0:7:3: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.871013] sd 6:0:7:3: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.876746] sd 6:0:8:0: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.882389] sd 6:0:8:0: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.890171] sd 6:0:8:0: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.895907] sd 6:0:8:1: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.901546] sd 6:0:8:1: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.909323] sd 6:0:8:1: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.915058] sd 6:0:8:2: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.920695] sd 6:0:8:2: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.928470] sd 6:0:8:2: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.934208] sd 6:0:8:3: Embedded Enclosure Device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.939864] sd 6:0:8:3: FIXME driver has no support for subenclosures (1) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.947644] sd 6:0:8:3: Failed to bind enclosure -12 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.960149] ses 6:0:4:0: Attached Enclosure device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 59.965701] ses 6:0:9:0: Attached Enclosure device Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.069000] sd 6:0:0:0: Attached scsi generic sg0 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.075162] sd 6:0:0:1: Attached scsi generic sg1 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.081315] sd 6:0:0:2: Attached scsi generic sg2 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.087471] sd 6:0:0:3: Attached scsi generic sg3 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.093623] sd 6:0:1:0: Attached scsi generic sg4 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.099762] sd 6:0:1:1: Attached scsi generic sg5 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.105923] sd 6:0:1:2: Attached scsi generic sg6 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.112077] sd 6:0:1:3: Attached scsi generic sg7 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.118232] sd 6:0:2:0: Attached scsi generic sg8 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.124383] sd 6:0:2:1: Attached scsi generic sg9 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.130530] sd 6:0:2:2: Attached scsi generic sg10 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.136779] sd 6:0:2:3: Attached scsi generic sg11 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.143029] sd 6:0:3:0: Attached scsi generic sg12 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.149270] sd 6:0:3:1: Attached scsi generic sg13 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.155529] sd 6:0:3:2: Attached scsi generic sg14 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.161775] sd 6:0:3:3: Attached scsi generic sg15 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.168028] ses 6:0:4:0: Attached scsi generic sg16 type 13 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.174472] sd 6:0:5:0: Attached scsi generic sg17 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.180718] sd 6:0:5:1: Attached scsi generic sg18 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.186967] sd 6:0:5:2: Attached scsi generic sg19 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.193215] sd 6:0:5:3: Attached scsi generic sg20 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.199470] sd 6:0:6:0: Attached scsi generic sg21 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.205721] sd 6:0:6:1: Attached scsi generic sg22 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.211984] sd 6:0:6:2: Attached scsi generic sg23 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.218231] sd 6:0:6:3: Attached scsi generic sg24 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.224510] sd 6:0:7:0: Attached scsi generic sg25 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.230765] sd 6:0:7:1: Attached scsi generic sg26 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.237025] sd 6:0:7:2: Attached scsi generic sg27 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.243272] sd 6:0:7:3: Attached scsi generic sg28 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.249525] sd 6:0:8:0: Attached scsi generic sg29 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.255776] sd 6:0:8:1: Attached scsi generic sg30 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.262037] sd 6:0:8:2: Attached scsi generic sg31 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.268290] sd 6:0:8:3: Attached scsi generic sg32 type 0 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.274540] ses 6:0:9:0: Attached scsi generic sg33 type 13 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.516573] device-mapper: uevent: version 1.0.3 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.522203] device-mapper: ioctl: 4.23.6-ioctl (2012-07-25) initialised: dm-devel@redhat.com Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.539226] device-mapper: multipath: version 1.5.0 loaded Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.641087] sd 6:0:0:0: rdac: LUN 0 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.647025] sd 6:0:0:1: rdac: LUN 1 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.653145] sd 6:0:0:2: rdac: LUN 2 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.659063] sd 6:0:0:3: rdac: LUN 3 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.665175] sd 6:0:1:0: rdac: LUN 0 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.671094] sd 6:0:1:1: rdac: LUN 1 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.679141] sd 6:0:1:2: rdac: LUN 2 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.685047] sd 6:0:1:3: rdac: LUN 3 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.691160] sd 6:0:2:0: rdac: LUN 0 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.697274] sd 6:0:2:1: rdac: LUN 1 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.703169] sd 6:0:2:2: rdac: LUN 2 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.709272] sd 6:0:2:3: rdac: LUN 3 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.715175] sd 6:0:3:0: rdac: LUN 0 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.721279] sd 6:0:3:1: rdac: LUN 1 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.727181] sd 6:0:3:2: rdac: LUN 2 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.733286] sd 6:0:3:3: rdac: LUN 3 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.739199] sd 6:0:5:0: rdac: LUN 0 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.745095] sd 6:0:5:1: rdac: LUN 1 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.751191] sd 6:0:5:2: rdac: LUN 2 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.757091] sd 6:0:5:3: rdac: LUN 3 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.763197] sd 6:0:6:0: rdac: LUN 0 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.769104] sd 6:0:6:1: rdac: LUN 1 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.775200] sd 6:0:6:2: rdac: LUN 2 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.781109] sd 6:0:6:3: rdac: LUN 3 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.787205] sd 6:0:7:0: rdac: LUN 0 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.793300] sd 6:0:7:1: rdac: LUN 1 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.799207] sd 6:0:7:2: rdac: LUN 2 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.805303] sd 6:0:7:3: rdac: LUN 3 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.811208] sd 6:0:8:0: rdac: LUN 0 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.817300] sd 6:0:8:1: rdac: LUN 1 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.823211] sd 6:0:8:2: rdac: LUN 2 (IOSHIP) (unowned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.829306] sd 6:0:8:3: rdac: LUN 3 (IOSHIP) (owned) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.835043] rdac: device handler registered Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 60.844829] device-mapper: multipath round-robin: version 1.0.0 loaded Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 62.572778] mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 62.873131] ip_tables: (C) 2000-2006 Netfilter Core Team Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 62.912230] nf_conntrack version 0.5.0 (16384 buckets, 65536 max) Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 63.096263] 802.1Q VLAN Support v1.8 Ben Greear Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 63.104045] All bugs added by David S. Miller Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 63.241817] ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 63.242411] ADDRCONF(NETDEV_UP): ib0: link is not ready Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 65.259332] ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22 Jul 17 17:45:00 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 67.868729] ipmi device interface Jul 17 17:45:01 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 69.242673] ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready Jul 17 17:45:02 atlas-oss4b4.ccs.ornl.gov kernel: [1918256.696967] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633497/real 1405633502] req@ffff880514c8d800 x1471904528872460/t0(0) o38->atlas2-MDT0000-lwp-OST0053@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:45:02 atlas-oss4b4.ccs.ornl.gov kernel: [1918256.782591] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:45:02 atlas-oss3h5.ccs.ornl.gov kernel: [1918278.598441] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633497/real 1405633502] req@ffff8802f4acb800 x1471904507807036/t0(0) o38->atlas2-MDT0000-lwp-OST003c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:45:02 atlas-oss3h5.ccs.ornl.gov kernel: [1918278.691477] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:45:02 atlas-oss3b1.ccs.ornl.gov kernel: [1918265.539722] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633497/real 1405633502] req@ffff88077ea76c00 x1471904516111192/t0(0) o38->atlas2-MDT0000-lwp-OST0008@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:45:02 atlas-oss3b1.ccs.ornl.gov kernel: [1918265.627010] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:45:03 atlas-oss3e6.ccs.ornl.gov kernel: [1918257.576497] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633497/real 1405633503] req@ffff880dec692400 x1471904520535080/t0(0) o38->atlas2-MDT0000-lwp-OST0025@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:45:03 atlas-oss3e6.ccs.ornl.gov kernel: [1918257.665645] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:45:04 atlas-oss3e2.ccs.ornl.gov kernel: [1918260.212242] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633497/real 1405633504] req@ffff8801edbd4000 x1471904513179640/t0(0) o38->atlas2-MDT0000-lwp-OST0021@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:45:04 atlas-oss3e2.ccs.ornl.gov kernel: [1918260.301818] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:45:05 atlas-oss4b2.ccs.ornl.gov kernel: [1918284.294558] Lustre: 27461:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633497/real 1405633505] req@ffff880ff0661c00 x1471904518026296/t0(0) o38->atlas2-MDT0000-lwp-OST0051@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:45:05 atlas-oss4b2.ccs.ornl.gov kernel: [1918284.386959] Lustre: 27461:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:45:05 atlas-oss4c1.ccs.ornl.gov kernel: [1918267.427279] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633497/real 1405633505] req@ffff880757926800 x1471904516097952/t0(0) o38->atlas2-MDT0000-lwp-OST0058@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:45:05 atlas-oss4c1.ccs.ornl.gov kernel: [1918267.518791] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:45:07 atlas-oss4g4.ccs.ornl.gov kernel: [1918278.318040] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633497/real 1405633507] req@ffff8806e69b8c00 x1471904515531116/t0(0) o38->atlas2-MDT0000-lwp-OST007b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:45:07 atlas-oss4g4.ccs.ornl.gov kernel: [1918278.409317] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:45:10 atlas-oss4f5.ccs.ornl.gov kernel: [1918264.796946] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633497/real 1405633510] req@ffff880bdc55d400 x1471904509391376/t0(0) o38->atlas2-MDT0000-lwp-OST0074@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:45:10 atlas-oss4f5.ccs.ornl.gov kernel: [1918264.889465] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:45:11 atlas-oss4e5.ccs.ornl.gov kernel: [1918269.071482] Lustre: 24644:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633497/real 1405633511] req@ffff880370145c00 x1471904515279652/t0(0) o38->atlas2-MDT0000-lwp-OST006c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:45:11 atlas-oss4e5.ccs.ornl.gov kernel: [1918269.158711] Lustre: 24644:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:45:11 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 79.267458] ib0: no IPv6 routers present Jul 17 17:45:14 atlas-oss3g7.ccs.ornl.gov kernel: [1916868.256999] Lustre: 13449:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633497/real 1405633514] req@ffff8806ecb66000 x1471904520422700/t0(0) o38->atlas2-MDT0000-lwp-OST00c6@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:45:14 atlas-oss3g7.ccs.ornl.gov kernel: [1916868.341679] Lustre: 13449:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:45:14 atlas-oss4g6.ccs.ornl.gov kernel: [1918280.765421] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633497/real 1405633514] req@ffff880706c27c00 x1471904507899140/t0(0) o38->atlas2-MDT0000-lwp-OST010d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:45:14 atlas-oss4g6.ccs.ornl.gov kernel: [1918280.851720] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:45:15 atlas-oss4g7.ccs.ornl.gov kernel: [1918293.808705] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633497/real 1405633515] req@ffff880a9951c400 x1471904508874024/t0(0) o38->atlas2-MDT0000-lwp-OST007e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633777 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:45:15 atlas-oss4g7.ccs.ornl.gov kernel: [1918293.899572] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:45:17 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 85.142742] netlink: 12 bytes leftover after parsing attributes. Jul 17 17:45:22 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 90.136274] Fusion MPT base driver 3.04.20 Jul 17 17:45:22 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 90.141044] Copyright (c) 1999-2008 LSI Corporation Jul 17 17:45:22 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 90.150739] Fusion MPT misc device (ioctl) driver 3.04.20 Jul 17 17:45:22 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 90.156985] mptctl: Registered with Fusion MPT base driver Jul 17 17:45:22 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 90.163321] mptctl: /dev/mptctl @ (major,minor=10,220) Jul 17 17:45:41 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: [ 108.220391] netlink: 12 bytes leftover after parsing attributes. Jul 17 17:45:41 atlas-mds3.ccs.ornl.gov gedi;lustre-mgmt1.ccs.ornl.gov;prod_lustre kernel: Kernel logging (proc) stopped. Jul 17 17:45:41 atlas-mds3.ccs.ornl.gov kernel: imklog 5.8.10, log source = /proc/kmsg started. Jul 17 17:45:42 atlas-mds3.ccs.ornl.gov kernel: [ 109.544267] ERST: NVRAM ERST Log Address Range is not implemented yet Jul 17 17:47:27 atlas-oss3e3.ccs.ornl.gov kernel: [1918399.972310] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff88046b33e800 x1471904510468932/t0(0) o38->atlas2-MDT0000-lwp-OST0022@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3e3.ccs.ornl.gov kernel: [1918400.059043] Lustre: 23062:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4i3.ccs.ornl.gov kernel: [1918395.530703] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8803e511b800 x1471904523160064/t0(0) o38->atlas2-MDT0000-lwp-OST008a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4i3.ccs.ornl.gov kernel: [1918395.615423] Lustre: 25949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:47:27 atlas-oss4d7.ccs.ornl.gov kernel: [1918404.021176] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880f61b3a400 x1471904514720464/t0(0) o38->atlas2-MDT0000-lwp-OST0066@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4d7.ccs.ornl.gov kernel: [1918404.113613] Lustre: 22274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4i4.ccs.ornl.gov kernel: [1918408.236704] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880e8224b000 x1471904516513912/t0(0) o38->atlas2-MDT0000-lwp-OST008b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4i4.ccs.ornl.gov kernel: [1918408.325428] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3f6.ccs.ornl.gov kernel: [805107.916316] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8809542a6000 x1473076856559064/t0(0) o38->atlas2-MDT0000-lwp-OST002d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3f6.ccs.ornl.gov kernel: [805108.005116] Lustre: 79665:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3c1.ccs.ornl.gov kernel: [1918405.464522] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8806ea9a5c00 x1471904511393452/t0(0) o38->atlas2-MDT0000-lwp-OST0010@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3c1.ccs.ornl.gov kernel: [1918405.554994] Lustre: 23675:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3i7.ccs.ornl.gov kernel: [1918400.481008] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880f5bbb5000 x1471904508389776/t0(0) o38->atlas2-MDT0000-lwp-OST0046@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3i7.ccs.ornl.gov kernel: [1918400.570740] Lustre: 24574:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:47:27 atlas-oss4f2.ccs.ornl.gov kernel: [1918406.908897] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880f5e5cfc00 x1471904508911164/t0(0) o38->atlas2-MDT0000-lwp-OST0071@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4f2.ccs.ornl.gov kernel: [1918406.998480] Lustre: 25720:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4d3.ccs.ornl.gov kernel: [1918399.807900] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880b85351400 x1471904516978744/t0(0) o38->atlas2-MDT0000-lwp-OST0062@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4d3.ccs.ornl.gov kernel: [1918399.900104] Lustre: 22325:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:47:27 atlas-oss4h4.ccs.ornl.gov kernel: [1918422.782441] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff88078e9bc800 x1471904512334840/t0(0) o38->atlas2-MDT0000-lwp-OST0083@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4h4.ccs.ornl.gov kernel: [1918422.867699] Lustre: 25828:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4c5.ccs.ornl.gov kernel: [1918399.089399] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff88075cd73400 x1471904510171704/t0(0) o38->atlas2-MDT0000-lwp-OST005c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4c5.ccs.ornl.gov kernel: [1918399.182607] Lustre: 26656:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4c3.ccs.ornl.gov kernel: [1918400.132578] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880bd9fb4400 x1471904512246556/t0(0) o38->atlas2-MDT0000-lwp-OST005a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4c3.ccs.ornl.gov kernel: [1918400.218918] Lustre: 24960:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3d6.ccs.ornl.gov kernel: [1918417.773491] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff88070767d400 x1471904511183860/t0(0) o38->atlas2-MDT0000-lwp-OST001d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3d6.ccs.ornl.gov kernel: [1918417.858456] Lustre: 25607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3g8.ccs.ornl.gov kernel: [1917001.414636] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880267bd1000 x1471904513602896/t0(0) o38->atlas2-MDT0000-lwp-OST0037@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3g8.ccs.ornl.gov kernel: [1917001.504496] Lustre: 13440:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3h8.ccs.ornl.gov kernel: [1918400.229758] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8802589b0c00 x1471904514773300/t0(0) o38->atlas2-MDT0000-lwp-OST003f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3h8.ccs.ornl.gov kernel: [1918400.320323] Lustre: 22176:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4a5.ccs.ornl.gov kernel: [1918422.370344] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880c24413c00 x1471904519454624/t0(0) o38->atlas2-MDT0000-lwp-OST004c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4a5.ccs.ornl.gov kernel: [1918422.455569] Lustre: 22122:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4a4.ccs.ornl.gov kernel: [1918404.921563] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880274371000 x1471904509591576/t0(0) o38->atlas2-MDT0000-lwp-OST004b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4a4.ccs.ornl.gov kernel: [1918405.014688] Lustre: 27348:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3e1.ccs.ornl.gov kernel: [1918406.138091] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880c2f073400 x1471904509694824/t0(0) o38->atlas2-MDT0000-lwp-OST0020@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3e1.ccs.ornl.gov kernel: [1918406.222914] Lustre: 23603:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3f5.ccs.ornl.gov kernel: [805095.279235] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880fff32ac00 x1473076856406772/t0(0) o38->atlas2-MDT0000-lwp-OST002c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3f5.ccs.ornl.gov kernel: [805095.364324] Lustre: 79198:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:47:27 atlas-oss4c8.ccs.ornl.gov kernel: [1918395.246257] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880456796c00 x1471904512197396/t0(0) o38->atlas2-MDT0000-lwp-OST005f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3f8.ccs.ornl.gov kernel: [805105.792632] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880d11638c00 x1473076861321420/t0(0) o38->atlas2-MDT0000-lwp-OST002f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4c8.ccs.ornl.gov kernel: [1918395.337621] Lustre: 22045:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3f8.ccs.ornl.gov kernel: [805105.877361] Lustre: 81086:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4e7.ccs.ornl.gov kernel: [1918400.791689] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880328b35000 x1471904522621992/t0(0) o38->atlas2-MDT0000-lwp-OST006e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4e7.ccs.ornl.gov kernel: [1918400.884755] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:47:27 atlas-oss4a2.ccs.ornl.gov kernel: [1918418.039694] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8806a0570400 x1471904510896188/t0(0) o38->atlas2-MDT0000-lwp-OST0049@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4a2.ccs.ornl.gov kernel: [1918418.134475] Lustre: 25778:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3g5.ccs.ornl.gov kernel: [1917005.001500] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff88011aee2000 x1471904518249564/t0(0) o38->atlas2-MDT0000-lwp-OST0034@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3g5.ccs.ornl.gov kernel: [1917005.086550] Lustre: 13556:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4c2.ccs.ornl.gov kernel: [1918407.340536] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880f1169c800 x1471904508571144/t0(0) o38->atlas2-MDT0000-lwp-OST0059@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4c2.ccs.ornl.gov kernel: [1918407.433017] Lustre: 25978:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4e1.ccs.ornl.gov kernel: [1918401.284241] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880d11a45400 x1471904515145232/t0(0) o38->atlas2-MDT0000-lwp-OST0068@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4e1.ccs.ornl.gov kernel: [1918401.369541] Lustre: 26041:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3h3.ccs.ornl.gov kernel: [1918399.033337] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8802ef18f400 x1471904512414084/t0(0) o38->atlas2-MDT0000-lwp-OST003a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3h3.ccs.ornl.gov kernel: [1918399.119591] Lustre: 26623:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3a1.ccs.ornl.gov kernel: [1918398.260257] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff88102f508800 x1471904525932240/t0(0) o38->atlas2-MDT0000-lwp-OST0000@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3a1.ccs.ornl.gov kernel: [1918398.349719] Lustre: 22431:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4d4.ccs.ornl.gov kernel: [1918423.812140] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880a3f3d4000 x1471904513679632/t0(0) o38->atlas2-MDT0000-lwp-OST0063@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4d4.ccs.ornl.gov kernel: [1918423.900075] Lustre: 27379:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4e2.ccs.ornl.gov kernel: [1918417.273104] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880974c05800 x1471904514146820/t0(0) o38->atlas2-MDT0000-lwp-OST0069@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4e2.ccs.ornl.gov kernel: [1918417.358784] Lustre: 22167:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:47:27 atlas-oss3h7.ccs.ornl.gov kernel: [1918410.171488] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8803c23d7c00 x1471904510333420/t0(0) o38->atlas2-MDT0000-lwp-OST003e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3h7.ccs.ornl.gov kernel: [1918410.258559] Lustre: 25081:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4g8.ccs.ornl.gov kernel: [1918402.532599] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff88086ba72c00 x1471904521747304/t0(0) o38->atlas2-MDT0000-lwp-OST007f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4g8.ccs.ornl.gov kernel: [1918402.624743] Lustre: 25550:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4h8.ccs.ornl.gov kernel: [1918394.386507] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8806cf117400 x1471904514267680/t0(0) o38->atlas2-MDT0000-lwp-OST0087@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4h8.ccs.ornl.gov kernel: [1918394.472570] Lustre: 24621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3f7.ccs.ornl.gov kernel: [804734.520843] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880736840000 x1473076863306668/t0(0) o38->atlas2-MDT0000-lwp-OST002e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3f7.ccs.ornl.gov kernel: [804734.613791] Lustre: 78666:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4b6.ccs.ornl.gov kernel: [1918401.066105] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8805fcc11400 x1471904511214044/t0(0) o38->atlas2-MDT0000-lwp-OST0055@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4b6.ccs.ornl.gov kernel: [1918401.157481] Lustre: 26089:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3b3.ccs.ornl.gov kernel: [1918396.428001] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8803c4ab2400 x1471904513820904/t0(0) o38->atlas2-MDT0000-lwp-OST000a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3b3.ccs.ornl.gov kernel: [1918396.516955] Lustre: 25585:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4b1.ccs.ornl.gov kernel: [1918407.967254] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff88060b09cc00 x1471904507949568/t0(0) o38->atlas2-MDT0000-lwp-OST0050@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4b1.ccs.ornl.gov kernel: [1918408.059496] Lustre: 25862:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4g3.ccs.ornl.gov kernel: [1918414.654987] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880871399800 x1471904517610708/t0(0) o38->atlas2-MDT0000-lwp-OST007a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4g3.ccs.ornl.gov kernel: [1918414.748238] Lustre: 25818:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3a5.ccs.ornl.gov kernel: [1918397.027068] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff88074fc7ac00 x1471904511738944/t0(0) o38->atlas2-MDT0000-lwp-OST0004@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3a5.ccs.ornl.gov kernel: [1918397.120254] Lustre: 22226:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:47:27 atlas-oss3a6.ccs.ornl.gov kernel: [1918406.136758] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8806bee34c00 x1471904512395120/t0(0) o38->atlas2-MDT0000-lwp-OST0005@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3a6.ccs.ornl.gov kernel: [1918406.228617] Lustre: 25773:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3i4.ccs.ornl.gov kernel: [1918409.412730] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880527a38800 x1471904510470856/t0(0) o38->atlas2-MDT0000-lwp-OST0043@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3i4.ccs.ornl.gov kernel: [1918409.501252] Lustre: 27264:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4f6.ccs.ornl.gov kernel: [1918398.311779] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880efd6f4c00 x1471904511709036/t0(0) o38->atlas2-MDT0000-lwp-OST0075@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4f6.ccs.ornl.gov kernel: [1918398.402347] Lustre: 22280:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3b5.ccs.ornl.gov kernel: [1918416.565657] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8802bfa1d000 x1471904514717444/t0(0) o38->atlas2-MDT0000-lwp-OST000c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3b5.ccs.ornl.gov kernel: [1918416.658640] Lustre: 25953:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3h2.ccs.ornl.gov kernel: [1918406.601009] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff881071f93800 x1471904518760984/t0(0) o38->atlas2-MDT0000-lwp-OST0039@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3h2.ccs.ornl.gov kernel: [1918406.686510] Lustre: 20916:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4b7.ccs.ornl.gov kernel: [1918397.612343] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880c27fca400 x1471904509377768/t0(0) o38->atlas2-MDT0000-lwp-OST0056@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4b7.ccs.ornl.gov kernel: [1918397.705686] Lustre: 23152:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3h4.ccs.ornl.gov kernel: [1918407.340480] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880447ccd800 x1471904509245740/t0(0) o38->atlas2-MDT0000-lwp-OST003b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3h4.ccs.ornl.gov kernel: [1918407.434058] Lustre: 25735:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4b8.ccs.ornl.gov kernel: [1918405.420864] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880d8c88b000 x1471904512564388/t0(0) o38->atlas2-MDT0000-lwp-OST0057@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4b8.ccs.ornl.gov kernel: [1918405.512970] Lustre: 25791:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3e7.ccs.ornl.gov kernel: [1918405.965224] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880e95aba000 x1471904508195688/t0(0) o38->atlas2-MDT0000-lwp-OST0026@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3e7.ccs.ornl.gov kernel: [1918406.057271] Lustre: 25876:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3h1.ccs.ornl.gov kernel: [1918392.281969] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff88067c191400 x1471904517766376/t0(0) o38->atlas2-MDT0000-lwp-OST0038@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3h1.ccs.ornl.gov kernel: [1918392.370160] Lustre: 22205:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4e8.ccs.ornl.gov kernel: [1918416.563253] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8804a218dc00 x1471904518473272/t0(0) o38->atlas2-MDT0000-lwp-OST006f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4e8.ccs.ornl.gov kernel: [1918416.650292] Lustre: 24591:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:47:27 atlas-oss4i8.ccs.ornl.gov kernel: [1918417.309540] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8801a7488000 x1471904520247604/t0(0) o38->atlas2-MDT0000-lwp-OST008f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4i8.ccs.ornl.gov kernel: [1918417.396394] Lustre: 26091:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4f1.ccs.ornl.gov kernel: [1918407.780703] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880287a49000 x1471904513371540/t0(0) o38->atlas2-MDT0000-lwp-OST0070@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4f1.ccs.ornl.gov kernel: [1918407.868901] Lustre: 25952:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3d8.ccs.ornl.gov kernel: [1918410.187155] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880f21d9c400 x1471904508711148/t0(0) o38->atlas2-MDT0000-lwp-OST001f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3d8.ccs.ornl.gov kernel: [1918410.275961] Lustre: 24571:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:47:27 atlas-oss3f2.ccs.ornl.gov kernel: [1918398.751554] Lustre: 25710:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880dee247400 x1471904513673000/t0(0) o38->atlas2-MDT0000-lwp-OST0029@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3f2.ccs.ornl.gov kernel: [1918398.842867] Lustre: 25710:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Jul 17 17:47:27 atlas-oss4b5.ccs.ornl.gov kernel: [1918421.218955] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8804c85a1c00 x1471904518366136/t0(0) o38->atlas2-MDT0000-lwp-OST0054@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4b5.ccs.ornl.gov kernel: [1918421.303996] Lustre: 25780:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4i6.ccs.ornl.gov kernel: [1918417.038101] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8804e8991800 x1471904510598056/t0(0) o38->atlas2-MDT0000-lwp-OST008d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4i6.ccs.ornl.gov kernel: [1918417.131289] Lustre: 25719:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4f4.ccs.ornl.gov kernel: [1918404.104344] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880534fc6c00 x1471904516980736/t0(0) o38->atlas2-MDT0000-lwp-OST0073@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4f4.ccs.ornl.gov kernel: [1918404.197491] Lustre: 22154:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3e5.ccs.ornl.gov kernel: [1918404.904728] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880b2e28c000 x1471904510655280/t0(0) o38->atlas2-MDT0000-lwp-OST0024@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3e5.ccs.ornl.gov kernel: [1918404.991024] Lustre: 25940:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4a6.ccs.ornl.gov kernel: [1918405.729813] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8803e5e61800 x1471904512503248/t0(0) o38->atlas2-MDT0000-lwp-OST004d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4a6.ccs.ornl.gov kernel: [1918405.819257] Lustre: 25589:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4d6.ccs.ornl.gov kernel: [1918403.756861] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8804448d5000 x1471904519610892/t0(0) o38->atlas2-MDT0000-lwp-OST0065@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4d6.ccs.ornl.gov kernel: [1918403.845985] Lustre: 25545:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4g1.ccs.ornl.gov kernel: [1918400.748048] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8806f3070400 x1471904508970872/t0(0) o38->atlas2-MDT0000-lwp-OST0078@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4g1.ccs.ornl.gov kernel: [1918400.835766] Lustre: 25807:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3f3.ccs.ornl.gov kernel: [1918417.221209] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8805b7432c00 x1471904510688564/t0(0) o38->atlas2-MDT0000-lwp-OST002a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3f3.ccs.ornl.gov kernel: [1918417.314795] Lustre: 26009:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3c7.ccs.ornl.gov kernel: [1918404.561426] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8806ba30bc00 x1471904510249152/t0(0) o38->atlas2-MDT0000-lwp-OST0016@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3c7.ccs.ornl.gov kernel: [1918404.647337] Lustre: 25749:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:47:27 atlas-oss3e4.ccs.ornl.gov kernel: [1918417.962481] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8801bc136c00 x1471904508538500/t0(0) o38->atlas2-MDT0000-lwp-OST0023@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3e4.ccs.ornl.gov kernel: [1918418.049098] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3d3.ccs.ornl.gov kernel: [1918410.535708] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff88012a7b8c00 x1471904510126196/t0(0) o38->atlas2-MDT0000-lwp-OST001a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3d3.ccs.ornl.gov kernel: [1918410.622948] Lustre: 25540:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4a3.ccs.ornl.gov kernel: [1918419.686934] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880358d4cc00 x1471904514689368/t0(0) o38->atlas2-MDT0000-lwp-OST004a@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4a3.ccs.ornl.gov kernel: [1918419.772555] Lustre: 28191:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3c8.ccs.ornl.gov kernel: [1918408.755851] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880640ef5000 x1471904507928520/t0(0) o38->atlas2-MDT0000-lwp-OST0017@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3c8.ccs.ornl.gov kernel: [1918408.840173] Lustre: 26025:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4a7.ccs.ornl.gov kernel: [1918415.383028] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff88068bcc1000 x1471904510936496/t0(0) o38->atlas2-MDT0000-lwp-OST004e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4a7.ccs.ornl.gov kernel: [1918415.468511] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4a1.ccs.ornl.gov kernel: [1918401.653581] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880eb5f72800 x1471904510346280/t0(0) o38->atlas2-MDT0000-lwp-OST0048@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4a1.ccs.ornl.gov kernel: [1918401.744806] Lustre: 25981:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4e6.ccs.ornl.gov kernel: [1918419.490584] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8810175fb400 x1471904508408596/t0(0) o38->atlas2-MDT0000-lwp-OST006d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4e6.ccs.ornl.gov kernel: [1918419.580157] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3h6.ccs.ornl.gov kernel: [1918404.099580] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880e54c8a000 x1471904512217000/t0(0) o38->atlas2-MDT0000-lwp-OST003d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3h6.ccs.ornl.gov kernel: [1918404.192308] Lustre: 25745:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3d2.ccs.ornl.gov kernel: [1918405.439530] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8810268a3000 x1471904511215124/t0(0) o38->atlas2-MDT0000-lwp-OST0019@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3d2.ccs.ornl.gov kernel: [1918405.527863] Lustre: 25972:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:47:27 atlas-oss3c4.ccs.ornl.gov kernel: [1918399.018374] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8806cc371c00 x1471904511833396/t0(0) o38->atlas2-MDT0000-lwp-OST0013@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3c4.ccs.ornl.gov kernel: [1918399.105294] Lustre: 25947:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3i6.ccs.ornl.gov kernel: [1918402.584502] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8804da48d400 x1471904509634480/t0(0) o38->atlas2-MDT0000-lwp-OST0045@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3i6.ccs.ornl.gov kernel: [1918402.677911] Lustre: 25701:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3g3.ccs.ornl.gov kernel: [1917001.132102] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880267c4a400 x1471904516977236/t0(0) o38->atlas2-MDT0000-lwp-OST0032@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3g3.ccs.ornl.gov kernel: [1917001.225580] Lustre: 13949:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:47:27 atlas-oss3a7.ccs.ornl.gov kernel: [1918401.874602] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8807c1875c00 x1471904518656104/t0(0) o38->atlas2-MDT0000-lwp-OST0006@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3a7.ccs.ornl.gov kernel: [1918401.960402] Lustre: 22076:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4d1.ccs.ornl.gov kernel: [1918399.766665] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880ad17bf000 x1471904516812184/t0(0) o38->atlas2-MDT0000-lwp-OST0060@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4d1.ccs.ornl.gov kernel: [1918399.859985] Lustre: 22248:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3i3.ccs.ornl.gov kernel: [1918399.216541] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8809de931400 x1471904518169748/t0(0) o38->atlas2-MDT0000-lwp-OST0042@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3i3.ccs.ornl.gov kernel: [1918399.306210] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3g6.ccs.ornl.gov kernel: [1917012.146795] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880960770400 x1471904523474304/t0(0) o38->atlas2-MDT0000-lwp-OST0035@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3g6.ccs.ornl.gov kernel: [1917012.237917] Lustre: 13982:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4h3.ccs.ornl.gov kernel: [1918416.463818] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880d195df000 x1471904509651320/t0(0) o38->atlas2-MDT0000-lwp-OST0082@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4h3.ccs.ornl.gov kernel: [1918416.549628] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3c3.ccs.ornl.gov kernel: [1918396.444574] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8806e5ab4c00 x1471904511051920/t0(0) o38->atlas2-MDT0000-lwp-OST0012@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3c3.ccs.ornl.gov kernel: [1918396.530634] Lustre: 22219:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:47:27 atlas-oss3d7.ccs.ornl.gov kernel: [1918404.633803] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880a21fdc800 x1471904512149924/t0(0) o38->atlas2-MDT0000-lwp-OST001e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3d7.ccs.ornl.gov kernel: [1918404.719266] Lustre: 22218:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4a8.ccs.ornl.gov kernel: [1918409.819658] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880f5af7c000 x1471904507751240/t0(0) o38->atlas2-MDT0000-lwp-OST004f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4i1.ccs.ornl.gov kernel: [1918403.881356] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880753d90800 x1471904513320204/t0(0) o38->atlas2-MDT0000-lwp-OST0088@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4a8.ccs.ornl.gov kernel: [1918409.905496] Lustre: 25951:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4i1.ccs.ornl.gov kernel: [1918403.968489] Lustre: 25767:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss4f8.ccs.ornl.gov kernel: [1918403.021302] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8801ddfb4c00 x1471904511806504/t0(0) o38->atlas2-MDT0000-lwp-OST0077@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4f8.ccs.ornl.gov kernel: [1918403.110388] Lustre: 22211:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3f1.ccs.ornl.gov kernel: [1918407.611202] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff881020570c00 x1471904508988904/t0(0) o38->atlas2-MDT0000-lwp-OST0028@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3f1.ccs.ornl.gov kernel: [1918407.704461] Lustre: 24208:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3b7.ccs.ornl.gov kernel: [1918408.668364] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8808e46e1000 x1471904519393116/t0(0) o38->atlas2-MDT0000-lwp-OST000e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3b7.ccs.ornl.gov kernel: [1918408.755782] Lustre: 26034:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:47:27 atlas-oss3i1.ccs.ornl.gov kernel: [1918403.941233] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8805b01e0400 x1471904519755880/t0(0) o38->atlas2-MDT0000-lwp-OST0040@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3i1.ccs.ornl.gov kernel: [1918404.033432] Lustre: 22126:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Jul 17 17:47:27 atlas-oss4d8.ccs.ornl.gov kernel: [1918416.873271] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff8801db3a1400 x1471904516223468/t0(0) o38->atlas2-MDT0000-lwp-OST0067@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss4d8.ccs.ornl.gov kernel: [1918416.962490] Lustre: 22112:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3b8.ccs.ornl.gov kernel: [1918418.000990] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880add1cc400 x1471904516407252/t0(0) o38->atlas2-MDT0000-lwp-OST000f@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3b8.ccs.ornl.gov kernel: [1918418.087983] Lustre: 22217:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Jul 17 17:47:27 atlas-oss3d5.ccs.ornl.gov kernel: [1918407.342475] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880d79189800 x1471904509642504/t0(0) o38->atlas2-MDT0000-lwp-OST001c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3d5.ccs.ornl.gov kernel: [1918407.432201] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:47:27 atlas-oss3b4.ccs.ornl.gov kernel: [1918415.521434] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633647/real 1405633647] req@ffff880916b53400 x1471904510549032/t0(0) o38->atlas2-MDT0000-lwp-OST000b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633927 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:47:27 atlas-oss3b4.ccs.ornl.gov kernel: [1918415.611515] Lustre: 26915:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:48:13 atlas-oss3i5.ccs.ornl.gov kernel: [704757.886835] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633693/real 1405633693] req@ffff8801ac13e000 x1473175721933808/t0(0) o38->atlas2-MDT0000-lwp-OST00d4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633973 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:48:13 atlas-oss3i5.ccs.ornl.gov kernel: [704757.988490] Lustre: 17765:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:48:13 atlas-oss3a8.ccs.ornl.gov kernel: [704758.814306] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633693/real 1405633693] req@ffff88087043fc00 x1473175720195392/t0(0) o38->atlas2-MDT0000-lwp-OST0097@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633973 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:48:13 atlas-oss3a8.ccs.ornl.gov kernel: [704758.912640] Lustre: 17729:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:48:14 atlas-oss3d4.ccs.ornl.gov kernel: [704758.372169] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633693/real 1405633693] req@ffff880784da6c00 x1473175723813188/t0(0) o38->atlas2-MDT0000-lwp-OST013b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405633973 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:48:14 atlas-oss3d4.ccs.ornl.gov kernel: [704758.465736] Lustre: 16385:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:49:57 atlas-oss4g5.ccs.ornl.gov kernel: [1918553.373116] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633796/real 1405633796] req@ffff8807c7da9400 x1471904510457244/t0(0) o38->atlas2-MDT0000-lwp-OST022c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4g5.ccs.ornl.gov kernel: [1918553.464526] Lustre: 26050:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss4e4.ccs.ornl.gov kernel: [1918561.979713] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633796/real 1405633796] req@ffff8804ea287800 x1471904515374816/t0(0) o38->atlas2-MDT0000-lwp-OST006b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4e4.ccs.ornl.gov kernel: [1918562.070302] Lustre: 22188:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss4g2.ccs.ornl.gov kernel: [1918564.065933] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633796/real 1405633796] req@ffff880b2d33b400 x1471904509229468/t0(0) o38->atlas2-MDT0000-lwp-OST0109@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4g2.ccs.ornl.gov kernel: [1918564.151874] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss4h7.ccs.ornl.gov kernel: [1918561.578894] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633796/real 1405633797] req@ffff88042de5d000 x1471904512615644/t0(0) o38->atlas2-MDT0000-lwp-OST0086@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4h7.ccs.ornl.gov kernel: [1918561.668861] Lustre: 26103:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:49:57 atlas-oss4d2.ccs.ornl.gov kernel: [1918558.393835] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff8808e9550400 x1471904512221456/t0(0) o38->atlas2-MDT0000-lwp-OST0181@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4d2.ccs.ornl.gov kernel: [1918558.484871] Lustre: 27429:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss4h6.ccs.ornl.gov kernel: [1918552.090916] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633796/real 1405633797] req@ffff880616100800 x1471904515532592/t0(0) o38->atlas2-MDT0000-lwp-OST0085@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4h6.ccs.ornl.gov kernel: [1918552.176359] Lustre: 25964:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss4i7.ccs.ornl.gov kernel: [1918566.601654] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff88061d3fb400 x1471904511375612/t0(0) o38->atlas2-MDT0000-lwp-OST023e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4i7.ccs.ornl.gov kernel: [1918566.690941] Lustre: 25149:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss4i2.ccs.ornl.gov kernel: [1918576.202921] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633796/real 1405633797] req@ffff8806837b5400 x1471904514635024/t0(0) o38->atlas2-MDT0000-lwp-OST0089@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4i2.ccs.ornl.gov kernel: [1918576.288638] Lustre: 25743:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss4b3.ccs.ornl.gov kernel: [1918547.601120] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff88049b098400 x1471904509181016/t0(0) o38->atlas2-MDT0000-lwp-OST00e2@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4b3.ccs.ornl.gov kernel: [1918547.688731] Lustre: 22295:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss4f7.ccs.ornl.gov kernel: [1918552.378933] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633796/real 1405633797] req@ffff88068dc6f400 x1471904510455544/t0(0) o38->atlas2-MDT0000-lwp-OST0076@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4f7.ccs.ornl.gov kernel: [1918552.471350] Lustre: 25973:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Jul 17 17:49:57 atlas-oss4h2.ccs.ornl.gov kernel: [1918574.574653] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633796/real 1405633797] req@ffff88022f028c00 x1471904508411628/t0(0) o38->atlas2-MDT0000-lwp-OST0081@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634076 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4h2.ccs.ornl.gov kernel: [1918574.662848] Lustre: 25996:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss4c4.ccs.ornl.gov kernel: [1918565.282898] Lustre: 25786:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff8806429f7800 x1471904510981548/t0(0) o38->atlas2-MDT0000-lwp-OST005b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4c4.ccs.ornl.gov kernel: [1918565.368368] Lustre: 25786:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss3f4.ccs.ornl.gov kernel: [1918555.592734] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff88025fdb0400 x1471904507778964/t0(0) o38->atlas2-MDT0000-lwp-OST002b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3f4.ccs.ornl.gov kernel: [1918555.684403] Lustre: 22151:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss4f3.ccs.ornl.gov kernel: [1918548.324196] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff88049eb44000 x1471904513977636/t0(0) o38->atlas2-MDT0000-lwp-OST0102@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4f3.ccs.ornl.gov kernel: [1918548.417653] Lustre: 25999:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss3g2.ccs.ornl.gov kernel: [1917150.269021] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff88018d72e400 x1471904509036608/t0(0) o38->atlas2-MDT0000-lwp-OST0031@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3g2.ccs.ornl.gov kernel: [1917150.355281] Lustre: 13994:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss3g1.ccs.ornl.gov kernel: [1917143.580622] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff88074982f000 x1471904510300116/t0(0) o38->atlas2-MDT0000-lwp-OST0030@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3g1.ccs.ornl.gov kernel: [1917143.673853] Lustre: 13831:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss3a2.ccs.ornl.gov kernel: [1918553.164633] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff880a347cdc00 x1471904510524768/t0(0) o38->atlas2-MDT0000-lwp-OST0001@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3a2.ccs.ornl.gov kernel: [1918553.257374] Lustre: 25621:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:49:57 atlas-oss4d5.ccs.ornl.gov kernel: [1918566.178350] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff880198d5dc00 x1471904519137308/t0(0) o38->atlas2-MDT0000-lwp-OST0064@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4d5.ccs.ornl.gov kernel: [1918566.267847] Lustre: 26088:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss4e3.ccs.ornl.gov kernel: [1918559.474343] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff880930acd800 x1471904515524208/t0(0) o38->atlas2-MDT0000-lwp-OST00fa@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4e3.ccs.ornl.gov kernel: [1918559.567248] Lustre: 27355:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss3i8.ccs.ornl.gov kernel: [1918567.579763] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff880d54d3a400 x1471904515355956/t0(0) o38->atlas2-MDT0000-lwp-OST0047@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3i8.ccs.ornl.gov kernel: [1918567.672394] Lustre: 22168:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss4i5.ccs.ornl.gov kernel: [1918561.627296] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff8802059e5400 x1471904512720892/t0(0) o38->atlas2-MDT0000-lwp-OST008c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4i5.ccs.ornl.gov kernel: [1918561.719628] Lustre: 25958:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss4h5.ccs.ornl.gov kernel: [1918556.188251] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff8805a8138400 x1471904509313948/t0(0) o38->atlas2-MDT0000-lwp-OST0084@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4h5.ccs.ornl.gov kernel: [1918556.274775] Lustre: 25669:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss3a4.ccs.ornl.gov kernel: [1918543.654929] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff880ce8fb2400 x1471904512539268/t0(0) o38->atlas2-MDT0000-lwp-OST0003@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3a4.ccs.ornl.gov kernel: [1918543.746462] Lustre: 22178:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss4h1.ccs.ornl.gov kernel: [1918547.567975] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff8805540f9800 x1471904511603608/t0(0) o38->atlas2-MDT0000-lwp-OST0080@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4h1.ccs.ornl.gov kernel: [1918547.653424] Lustre: 25859:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss3d1.ccs.ornl.gov kernel: [1918546.355194] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff8803c9d9a800 x1471904515778796/t0(0) o38->atlas2-MDT0000-lwp-OST0018@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3d1.ccs.ornl.gov kernel: [1918546.445719] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss3e8.ccs.ornl.gov kernel: [1918553.817246] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff880495675400 x1471904515140068/t0(0) o38->atlas2-MDT0000-lwp-OST0027@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3e8.ccs.ornl.gov kernel: [1918553.910093] Lustre: 25784:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss3b6.ccs.ornl.gov kernel: [1918551.927799] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff880265bdb000 x1471904512275864/t0(0) o38->atlas2-MDT0000-lwp-OST000d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3a3.ccs.ornl.gov kernel: [1918550.937972] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff880d5432a400 x1471904526380400/t0(0) o38->atlas2-MDT0000-lwp-OST0002@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3b6.ccs.ornl.gov kernel: [1918552.021196] Lustre: 25955:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss3a3.ccs.ornl.gov kernel: [1918551.023569] Lustre: 27237:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss3b2.ccs.ornl.gov kernel: [1918553.706348] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff8807758ab400 x1471904511917492/t0(0) o38->atlas2-MDT0000-lwp-OST0099@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3b2.ccs.ornl.gov kernel: [1918553.797317] Lustre: 25274:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss3g4.ccs.ornl.gov kernel: [1917164.247311] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff8805cc2d9000 x1471904510791852/t0(0) o38->atlas2-MDT0000-lwp-OST00c3@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3g4.ccs.ornl.gov kernel: [1917164.334024] Lustre: 13607:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jul 17 17:49:57 atlas-oss3c2.ccs.ornl.gov kernel: [1918555.779443] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff880135f42800 x1471904511648424/t0(0) o38->atlas2-MDT0000-lwp-OST00a1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3c2.ccs.ornl.gov kernel: [1918555.867842] Lustre: 25857:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss3c5.ccs.ornl.gov kernel: [1918560.349524] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff880c1cb21c00 x1471904512466612/t0(0) o38->atlas2-MDT0000-lwp-OST00a4@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3c5.ccs.ornl.gov kernel: [1918560.438604] Lustre: 25796:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss3c6.ccs.ornl.gov kernel: [1918545.988362] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff8806247fc800 x1471904508187360/t0(0) o38->atlas2-MDT0000-lwp-OST0015@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3c6.ccs.ornl.gov kernel: [1918546.077059] Lustre: 24535:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss4c6.ccs.ornl.gov kernel: [1918554.673119] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff880f51b4b400 x1471904517937708/t0(0) o38->atlas2-MDT0000-lwp-OST005d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss4c6.ccs.ornl.gov kernel: [1918554.762954] Lustre: 22182:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:49:57 atlas-oss3i2.ccs.ornl.gov kernel: [1918560.113856] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1405633797/real 1405633797] req@ffff88029b065800 x1471904514016748/t0(0) o38->atlas2-MDT0000-lwp-OST00d1@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634077 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:49:57 atlas-oss3i2.ccs.ornl.gov kernel: [1918560.202973] Lustre: 25711:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Jul 17 17:52:12 atlas-mds3.ccs.ornl.gov kernel: [ 499.909895] LNet: HW CPU cores: 24, npartitions: 4 Jul 17 17:52:12 atlas-mds3.ccs.ornl.gov kernel: [ 499.918464] alg: No test for crc32 (crc32-table) Jul 17 17:52:12 atlas-mds3.ccs.ornl.gov kernel: [ 499.923782] alg: No test for adler32 (adler32-zlib) Jul 17 17:52:12 atlas-mds3.ccs.ornl.gov kernel: [ 499.929389] alg: No test for crc32 (crc32-pclmul) Jul 17 17:52:16 atlas-mds3.ccs.ornl.gov kernel: [ 503.955767] padlock: VIA PadLock Hash Engine not detected. Jul 17 17:52:20 atlas-mds3.ccs.ornl.gov kernel: [ 508.180784] LNet: Added LNI 10.36.226.74@o2ib [63/2560/0/180] Jul 17 17:52:21 atlas-mds3.ccs.ornl.gov kernel: [ 508.662783] LNet: Added LNI 10.36.226.74@o2ib200 [63/2560/0/180] Jul 17 17:52:21 atlas-mds3.ccs.ornl.gov kernel: [ 508.843289] Lustre: Lustre: Build Version: 2.4.3-gd00e4d6-CHANGED-2.6.32-358.23.2.el6.atlas.x86_64 Jul 17 17:52:23 atlas-mds3.ccs.ornl.gov kernel: [ 510.573038] LDISKFS-fs (dm-0): external journal device major/minor numbers have changed Jul 17 17:52:57 atlas-mds3.ccs.ornl.gov kernel: [ 544.895899] LDISKFS-fs (dm-0): recovery complete Jul 17 17:52:57 atlas-mds3.ccs.ornl.gov kernel: [ 544.959308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: Jul 17 17:52:59 atlas-mds3.ccs.ornl.gov kernel: [ 546.612526] Lustre: atlas2-MDT0000: used disk, loading Jul 17 17:53:05 atlas-mds3.ccs.ornl.gov kernel: [ 553.118451] LustreError: 11-0: atlas2-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. Jul 17 17:53:07 atlas-mds3.ccs.ornl.gov kernel: [ 555.153833] Lustre: atlas2-MDT0000: Will be in recovery for at least 30:00, or until 20156 clients reconnect Jul 17 17:53:11 atlas-mds3.ccs.ornl.gov kernel: [ 558.523139] LustreError: 15444:0:(mdt_open.c:1504:mdt_reint_open()) @@@ OPEN & CREAT not in open replay/by_fid. req@ffff881e84bf5850 x1473208624487752/t0(108775227966) o101->3d93993d-b972-8b87-5d44-4a9fec67bd60@10.38.144.124@o2ib4:0/0 lens 24736/3512 e 0 to 0 dl 1405634308 ref 1 fl Interpret:/4/0 rc 0/0 Jul 17 17:53:11 atlas-mds3.ccs.ornl.gov kernel: [ 558.554344] LustreError: 15444:0:(mdt_open.c:1504:mdt_reint_open()) @@@ OPEN & CREAT not in open replay/by_fid. req@ffff883fff07f000 x1473208624487756/t0(108775228599) o101->3d93993d-b972-8b87-5d44-4a9fec67bd60@10.38.144.124@o2ib4:0/0 lens 24736/3512 e 0 to 0 dl 1405634308 ref 1 fl Interpret:/4/0 rc 0/0 Jul 17 17:53:12 atlas-mds3.ccs.ornl.gov kernel: [ 559.402333] LustreError: 15746:0:(mdt_open.c:1504:mdt_reint_open()) @@@ OPEN & CREAT not in open replay/by_fid. req@ffff88400c097c00 x1473208624494624/t0(108775228386) o101->00be5d82-eb63-fe05-b08e-259511ca97c5@10.38.144.136@o2ib4:0/0 lens 24736/3512 e 0 to 0 dl 1405634308 ref 1 fl Interpret:/4/0 rc 0/0 Jul 17 17:53:12 atlas-mds3.ccs.ornl.gov kernel: [ 559.432987] LustreError: 15746:0:(mdt_open.c:1504:mdt_reint_open()) Skipped 22 previous similar messages Jul 17 17:53:13 atlas-oss3i5.ccs.ornl.gov kernel: [705057.965345] LustreError: 167-0: atlas2-MDT0000-lwp-OST00d4: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:53:13 atlas-oss3i5.ccs.ornl.gov kernel: [705058.002837] Lustre: atlas2-MDT0000-lwp-OST00d4: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:53:13 atlas-oss3a8.ccs.ornl.gov kernel: [705058.891722] LustreError: 167-0: atlas2-MDT0000-lwp-OST0097: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:53:13 atlas-oss3a8.ccs.ornl.gov kernel: [705058.925823] Lustre: atlas2-MDT0000-lwp-OST0097: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:53:13 atlas-oss3d4.ccs.ornl.gov kernel: [705058.454132] LustreError: 167-0: atlas2-MDT0000-lwp-OST013b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:53:14 atlas-oss3d4.ccs.ornl.gov kernel: [705058.488988] Lustre: atlas2-MDT0000-lwp-OST013b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:53:14 atlas-mds3.ccs.ornl.gov kernel: [ 561.773787] LustreError: 15513:0:(mdt_open.c:1504:mdt_reint_open()) @@@ OPEN & CREAT not in open replay/by_fid. req@ffff881e6d5f5c00 x1473208622167864/t0(108775228078) o101->cbed14be-5349-4597-c47e-95e1419b3de4@10.38.145.14@o2ib4:0/0 lens 24736/3512 e 0 to 0 dl 1405634311 ref 1 fl Interpret:/4/0 rc 0/0 Jul 17 17:53:14 atlas-mds3.ccs.ornl.gov kernel: [ 561.804338] LustreError: 15513:0:(mdt_open.c:1504:mdt_reint_open()) Skipped 11 previous similar messages Jul 17 17:54:12 atlas-mds3.ccs.ornl.gov kernel: [ 619.798518] LustreError: 15958:0:(mdt_open.c:1504:mdt_reint_open()) @@@ OPEN & CREAT not in open replay/by_fid. req@ffff881e71bdb400 x1472462044726020/t0(108775227829) o101->bb89d2d6-5fee-a494-496a-39843fdf2e92@10.38.144.47@o2ib4:0/0 lens 24736/3512 e 0 to 0 dl 1405634369 ref 1 fl Interpret:/4/0 rc 0/0 Jul 17 17:54:12 atlas-mds3.ccs.ornl.gov kernel: [ 619.828991] LustreError: 15958:0:(mdt_open.c:1504:mdt_reint_open()) Skipped 47 previous similar messages Jul 17 17:54:17 atlas-mds3.ccs.ornl.gov kernel: [ 625.245468] LustreError: 15633:0:(mdt_open.c:1504:mdt_reint_open()) @@@ OPEN & CREAT not in open replay/by_fid. req@ffff88400d041800 x1472463073884684/t0(108775228063) o101->b0b4a9f1-a2f9-b0b4-69c5-e321aa61f780@10.38.144.144@o2ib4:0/0 lens 24736/3512 e 0 to 0 dl 1405634374 ref 1 fl Interpret:/4/0 rc 0/0 Jul 17 17:54:17 atlas-mds3.ccs.ornl.gov kernel: [ 625.276038] LustreError: 15633:0:(mdt_open.c:1504:mdt_reint_open()) Skipped 2776 previous similar messages Jul 17 17:54:56 atlas-oss4g5.ccs.ornl.gov kernel: [1918853.450977] LustreError: 167-0: atlas2-MDT0000-lwp-OST03dc: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:56 atlas-oss4g5.ccs.ornl.gov kernel: [1918853.485785] LustreError: 167-0: atlas2-MDT0000-lwp-OST02bc: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:56 atlas-oss4e4.ccs.ornl.gov kernel: [1918862.059905] LustreError: 167-0: atlas2-MDT0000-lwp-OST018b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g2.ccs.ornl.gov kernel: [1918864.146520] LustreError: 167-0: atlas2-MDT0000-lwp-OST03d9: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h7.ccs.ornl.gov kernel: [1918861.657948] LustreError: 167-0: atlas2-MDT0000-lwp-OST03e6: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h7.ccs.ornl.gov kernel: [1918861.691781] LustreError: 167-0: atlas2-MDT0000-lwp-OST02c6: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d2.ccs.ornl.gov kernel: [1918858.473216] LustreError: 167-0: atlas2-MDT0000-lwp-OST0331: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i7.ccs.ornl.gov kernel: [1918866.681670] LustreError: 167-0: atlas2-MDT0000-lwp-OST01ae: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h6.ccs.ornl.gov kernel: [1918852.168597] LustreError: 167-0: atlas2-MDT0000-lwp-OST03e5: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i2.ccs.ornl.gov kernel: [1918876.284067] LustreError: 167-0: atlas2-MDT0000-lwp-OST01a9: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i2.ccs.ornl.gov kernel: [1918876.324426] LustreError: 167-0: atlas2-MDT0000-lwp-OST0239: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4b3.ccs.ornl.gov kernel: [1918847.677907] LustreError: 167-0: atlas2-MDT0000-lwp-OST00e2: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4b3.ccs.ornl.gov kernel: [1918847.719928] LustreError: 167-0: atlas2-MDT0000-lwp-OST0172: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f7.ccs.ornl.gov kernel: [1918852.457176] LustreError: 167-0: atlas2-MDT0000-lwp-OST02b6: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h2.ccs.ornl.gov kernel: [1918874.655511] LustreError: 167-0: atlas2-MDT0000-lwp-OST0111: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h2.ccs.ornl.gov kernel: [1918874.697586] LustreError: 167-0: atlas2-MDT0000-lwp-OST03e1: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4b2.ccs.ornl.gov kernel: [1918876.251608] LustreError: 167-0: atlas2-MDT0000-lwp-OST0291: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g5.ccs.ornl.gov kernel: [1918853.486401] Lustre: atlas2-MDT0000-lwp-OST03dc: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3e3.ccs.ornl.gov kernel: [1918850.105270] LustreError: 167-0: atlas2-MDT0000-lwp-OST02f2: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e4.ccs.ornl.gov kernel: [1918862.093974] LustreError: 167-0: atlas2-MDT0000-lwp-OST021b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e4.ccs.ornl.gov kernel: [1918862.094588] Lustre: atlas2-MDT0000-lwp-OST018b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g2.ccs.ornl.gov kernel: [1918864.185705] LustreError: 167-0: atlas2-MDT0000-lwp-OST02b9: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g2.ccs.ornl.gov kernel: [1918864.186343] Lustre: atlas2-MDT0000-lwp-OST03d9: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4h7.ccs.ornl.gov kernel: [1918861.692390] Lustre: atlas2-MDT0000-lwp-OST03e6: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4i3.ccs.ornl.gov kernel: [1918845.662040] LustreError: 167-0: atlas2-MDT0000-lwp-OST023a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d2.ccs.ornl.gov kernel: [1918858.507294] LustreError: 167-0: atlas2-MDT0000-lwp-OST03c1: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d2.ccs.ornl.gov kernel: [1918858.507889] Lustre: atlas2-MDT0000-lwp-OST0331: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4d7.ccs.ornl.gov kernel: [1918854.157116] LustreError: 167-0: atlas2-MDT0000-lwp-OST00f6: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d7.ccs.ornl.gov kernel: [1918854.194049] LustreError: 167-0: atlas2-MDT0000-lwp-OST0186: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h5.ccs.ornl.gov kernel: [1918873.196176] LustreError: 167-0: atlas2-MDT0000-lwp-OST027c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i4.ccs.ornl.gov kernel: [1918858.372605] LustreError: 167-0: atlas2-MDT0000-lwp-OST02cb: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h5.ccs.ornl.gov kernel: [1918873.232475] LustreError: 167-0: atlas2-MDT0000-lwp-OST030c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i7.ccs.ornl.gov kernel: [1918866.715690] LustreError: 167-0: atlas2-MDT0000-lwp-OST023e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i4.ccs.ornl.gov kernel: [1918858.407325] LustreError: 167-0: atlas2-MDT0000-lwp-OST035b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i7.ccs.ornl.gov kernel: [1918866.716274] Lustre: atlas2-MDT0000-lwp-OST01ae: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g5.ccs.ornl.gov kernel: [1918853.566519] Lustre: atlas2-MDT0000-lwp-OST02bc: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g5.ccs.ornl.gov kernel: [1918853.596210] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3f6.ccs.ornl.gov kernel: [805558.050865] LustreError: 167-0: atlas2-MDT0000-lwp-OST01dd: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3f6.ccs.ornl.gov kernel: [805558.085701] LustreError: 167-0: atlas2-MDT0000-lwp-OST014d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e4.ccs.ornl.gov kernel: [1918862.174551] Lustre: atlas2-MDT0000-lwp-OST021b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4e4.ccs.ornl.gov kernel: [1918862.204360] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4h6.ccs.ornl.gov kernel: [1918852.208565] LustreError: 167-0: atlas2-MDT0000-lwp-OST02c5: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h6.ccs.ornl.gov kernel: [1918852.209151] Lustre: atlas2-MDT0000-lwp-OST03e5: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c1.ccs.ornl.gov kernel: [1918855.599801] LustreError: 167-0: atlas2-MDT0000-lwp-OST0370: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3c1.ccs.ornl.gov kernel: [1918855.634375] LustreError: 167-0: atlas2-MDT0000-lwp-OST02e0: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3i7.ccs.ornl.gov kernel: [1918850.614799] LustreError: 167-0: atlas2-MDT0000-lwp-OST0166: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3i7.ccs.ornl.gov kernel: [1918850.649747] LustreError: 167-0: atlas2-MDT0000-lwp-OST01f6: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i2.ccs.ornl.gov kernel: [1918876.325066] Lustre: atlas2-MDT0000-lwp-OST01a9: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4f2.ccs.ornl.gov kernel: [1918857.043971] LustreError: 167-0: atlas2-MDT0000-lwp-OST02b1: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f2.ccs.ornl.gov kernel: [1918857.078271] LustreError: 167-0: atlas2-MDT0000-lwp-OST0341: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4b3.ccs.ornl.gov kernel: [1918847.720556] Lustre: atlas2-MDT0000-lwp-OST00e2: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4f7.ccs.ornl.gov kernel: [1918852.494173] LustreError: 167-0: atlas2-MDT0000-lwp-OST0196: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f7.ccs.ornl.gov kernel: [1918852.494778] Lustre: atlas2-MDT0000-lwp-OST02b6: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4d3.ccs.ornl.gov kernel: [1918849.941749] LustreError: 167-0: atlas2-MDT0000-lwp-OST0182: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d3.ccs.ornl.gov kernel: [1918849.978672] LustreError: 167-0: atlas2-MDT0000-lwp-OST0212: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c4.ccs.ornl.gov kernel: [1918865.362846] LustreError: 167-0: atlas2-MDT0000-lwp-OST03bb: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c4.ccs.ornl.gov kernel: [1918865.402505] LustreError: 167-0: atlas2-MDT0000-lwp-OST032b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g2.ccs.ornl.gov kernel: [1918864.266549] Lustre: atlas2-MDT0000-lwp-OST02b9: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4h2.ccs.ornl.gov kernel: [1918874.698213] Lustre: atlas2-MDT0000-lwp-OST0111: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g2.ccs.ornl.gov kernel: [1918864.304779] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4h7.ccs.ornl.gov kernel: [1918861.772263] Lustre: atlas2-MDT0000-lwp-OST02c6: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4h7.ccs.ornl.gov kernel: [1918861.810351] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss4h4.ccs.ornl.gov kernel: [1918872.920277] LustreError: 167-0: atlas2-MDT0000-lwp-OST0353: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h4.ccs.ornl.gov kernel: [1918872.957638] LustreError: 167-0: atlas2-MDT0000-lwp-OST03e3: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c5.ccs.ornl.gov kernel: [1918849.222667] LustreError: 167-0: atlas2-MDT0000-lwp-OST029c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c5.ccs.ornl.gov kernel: [1918849.261831] LustreError: 167-0: atlas2-MDT0000-lwp-OST03bc: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c3.ccs.ornl.gov kernel: [1918850.264027] LustreError: 167-0: atlas2-MDT0000-lwp-OST00ea: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c3.ccs.ornl.gov kernel: [1918850.297849] LustreError: 167-0: atlas2-MDT0000-lwp-OST032a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d2.ccs.ornl.gov kernel: [1918858.588113] Lustre: atlas2-MDT0000-lwp-OST03c1: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4d2.ccs.ornl.gov kernel: [1918858.626381] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4i7.ccs.ornl.gov kernel: [1918866.796343] Lustre: atlas2-MDT0000-lwp-OST023e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4i7.ccs.ornl.gov kernel: [1918866.834980] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3g8.ccs.ornl.gov kernel: [1917451.549119] LustreError: 167-0: atlas2-MDT0000-lwp-OST0157: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3g8.ccs.ornl.gov kernel: [1917451.582845] LustreError: 167-0: atlas2-MDT0000-lwp-OST01e7: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i2.ccs.ornl.gov kernel: [1918876.404997] Lustre: atlas2-MDT0000-lwp-OST0239: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4i2.ccs.ornl.gov kernel: [1918876.434752] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3h8.ccs.ornl.gov kernel: [1918850.362167] LustreError: 167-0: atlas2-MDT0000-lwp-OST039f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h8.ccs.ornl.gov kernel: [1918850.397694] LustreError: 167-0: atlas2-MDT0000-lwp-OST030f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4b2.ccs.ornl.gov kernel: [1918876.293830] LustreError: 167-0: atlas2-MDT0000-lwp-OST0321: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4b2.ccs.ornl.gov kernel: [1918876.294448] Lustre: atlas2-MDT0000-lwp-OST0291: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f4.ccs.ornl.gov kernel: [1918855.670981] LustreError: 167-0: atlas2-MDT0000-lwp-OST026b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3f4.ccs.ornl.gov kernel: [1918855.706362] LustreError: 167-0: atlas2-MDT0000-lwp-OST02fb: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4b3.ccs.ornl.gov kernel: [1918847.800776] Lustre: atlas2-MDT0000-lwp-OST0172: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b3.ccs.ornl.gov kernel: [1918847.830686] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3d6.ccs.ornl.gov kernel: [1918867.910877] LustreError: 167-0: atlas2-MDT0000-lwp-OST037d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4a4.ccs.ornl.gov kernel: [1918855.055184] LustreError: 167-0: atlas2-MDT0000-lwp-OST031b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4a4.ccs.ornl.gov kernel: [1918855.093629] LustreError: 167-0: atlas2-MDT0000-lwp-OST028b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e3.ccs.ornl.gov kernel: [1918850.146866] LustreError: 167-0: atlas2-MDT0000-lwp-OST0262: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e3.ccs.ornl.gov kernel: [1918850.147507] Lustre: atlas2-MDT0000-lwp-OST02f2: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4f7.ccs.ornl.gov kernel: [1918852.575018] Lustre: atlas2-MDT0000-lwp-OST0196: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4f7.ccs.ornl.gov kernel: [1918852.613215] Lustre: Skipped 2 previous similar messages Jul 17 17:54:57 atlas-oss4h2.ccs.ornl.gov kernel: [1918874.778482] Lustre: atlas2-MDT0000-lwp-OST03e1: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4h2.ccs.ornl.gov kernel: [1918874.808306] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4a5.ccs.ornl.gov kernel: [1918872.509669] LustreError: 167-0: atlas2-MDT0000-lwp-OST016c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4a5.ccs.ornl.gov kernel: [1918872.550420] LustreError: 167-0: atlas2-MDT0000-lwp-OST01fc: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e1.ccs.ornl.gov kernel: [1918856.273384] LustreError: 167-0: atlas2-MDT0000-lwp-OST00b0: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e1.ccs.ornl.gov kernel: [1918856.311717] LustreError: 167-0: atlas2-MDT0000-lwp-OST0140: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c8.ccs.ornl.gov kernel: [1918845.378450] LustreError: 167-0: atlas2-MDT0000-lwp-OST020f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i3.ccs.ornl.gov kernel: [1918845.701998] LustreError: 167-0: atlas2-MDT0000-lwp-OST02ca: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i3.ccs.ornl.gov kernel: [1918845.702586] Lustre: atlas2-MDT0000-lwp-OST023a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4e7.ccs.ornl.gov kernel: [1918850.923606] LustreError: 167-0: atlas2-MDT0000-lwp-OST033e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3f5.ccs.ornl.gov kernel: [805545.410843] LustreError: 167-0: atlas2-MDT0000-lwp-OST014c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3f5.ccs.ornl.gov kernel: [805545.450929] LustreError: 167-0: atlas2-MDT0000-lwp-OST01dc: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4a2.ccs.ornl.gov kernel: [1918868.176105] LustreError: 167-0: atlas2-MDT0000-lwp-OST0289: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h6.ccs.ornl.gov kernel: [1918852.289429] Lustre: atlas2-MDT0000-lwp-OST02c5: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4h6.ccs.ornl.gov kernel: [1918852.289432] Lustre: atlas2-MDT0000-lwp-OST0355: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4h6.ccs.ornl.gov kernel: [1918852.357796] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss4d7.ccs.ornl.gov kernel: [1918854.194640] Lustre: atlas2-MDT0000-lwp-OST00f6: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f8.ccs.ornl.gov kernel: [805555.925206] LustreError: 167-0: atlas2-MDT0000-lwp-OST026f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3f8.ccs.ornl.gov kernel: [805555.966170] LustreError: 167-0: atlas2-MDT0000-lwp-OST02ff: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i4.ccs.ornl.gov kernel: [1918858.407959] Lustre: atlas2-MDT0000-lwp-OST02cb: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h5.ccs.ornl.gov kernel: [1918873.233121] Lustre: atlas2-MDT0000-lwp-OST027c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f6.ccs.ornl.gov kernel: [805558.086332] Lustre: atlas2-MDT0000-lwp-OST01dd: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b2.ccs.ornl.gov kernel: [1918876.374476] Lustre: atlas2-MDT0000-lwp-OST0321: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b2.ccs.ornl.gov kernel: [1918876.412537] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4c2.ccs.ornl.gov kernel: [1918857.475161] LustreError: 167-0: atlas2-MDT0000-lwp-OST03b9: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3c1.ccs.ornl.gov kernel: [1918855.635039] Lustre: atlas2-MDT0000-lwp-OST0370: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i7.ccs.ornl.gov kernel: [1918850.650358] Lustre: atlas2-MDT0000-lwp-OST0166: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3e3.ccs.ornl.gov kernel: [1918850.227848] Lustre: atlas2-MDT0000-lwp-OST0262: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g5.ccs.ornl.gov kernel: [1917455.134731] LustreError: 167-0: atlas2-MDT0000-lwp-OST0274: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e3.ccs.ornl.gov kernel: [1918850.265728] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4f2.ccs.ornl.gov kernel: [1918857.078913] Lustre: atlas2-MDT0000-lwp-OST02b1: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4d3.ccs.ornl.gov kernel: [1918849.979298] Lustre: atlas2-MDT0000-lwp-OST0182: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4d7.ccs.ornl.gov kernel: [1918854.274767] Lustre: atlas2-MDT0000-lwp-OST0186: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4d7.ccs.ornl.gov kernel: [1918854.304435] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4c4.ccs.ornl.gov kernel: [1918865.403152] Lustre: atlas2-MDT0000-lwp-OST03bb: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4i3.ccs.ornl.gov kernel: [1918845.782651] Lustre: atlas2-MDT0000-lwp-OST02ca: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4e1.ccs.ornl.gov kernel: [1918851.417263] LustreError: 167-0: atlas2-MDT0000-lwp-OST02a8: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i3.ccs.ornl.gov kernel: [1918845.820851] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss4e1.ccs.ornl.gov kernel: [1918851.456720] LustreError: 167-0: atlas2-MDT0000-lwp-OST0338: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h4.ccs.ornl.gov kernel: [1918872.958282] Lustre: atlas2-MDT0000-lwp-OST0353: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4c5.ccs.ornl.gov kernel: [1918849.262482] Lustre: atlas2-MDT0000-lwp-OST029c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4c3.ccs.ornl.gov kernel: [1918850.298460] Lustre: atlas2-MDT0000-lwp-OST00ea: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c1.ccs.ornl.gov kernel: [1918855.714993] Lustre: atlas2-MDT0000-lwp-OST02e0: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c1.ccs.ornl.gov kernel: [1918855.744788] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4i4.ccs.ornl.gov kernel: [1918858.487784] Lustre: atlas2-MDT0000-lwp-OST035b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h5.ccs.ornl.gov kernel: [1918873.313416] Lustre: atlas2-MDT0000-lwp-OST039c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4i4.ccs.ornl.gov kernel: [1918858.526092] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3h5.ccs.ornl.gov kernel: [1918873.351468] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3g8.ccs.ornl.gov kernel: [1917451.583481] Lustre: atlas2-MDT0000-lwp-OST0157: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4d4.ccs.ornl.gov kernel: [1918873.949780] LustreError: 167-0: atlas2-MDT0000-lwp-OST03c3: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d4.ccs.ornl.gov kernel: [1918873.983571] LustreError: 167-0: atlas2-MDT0000-lwp-OST0333: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g8.ccs.ornl.gov kernel: [1918852.665860] LustreError: 167-0: atlas2-MDT0000-lwp-OST034f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3f6.ccs.ornl.gov kernel: [805558.166268] Lustre: atlas2-MDT0000-lwp-OST014d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f6.ccs.ornl.gov kernel: [805558.204421] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3a1.ccs.ornl.gov kernel: [1918848.393215] LustreError: 167-0: atlas2-MDT0000-lwp-OST0090: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3a1.ccs.ornl.gov kernel: [1918848.428923] LustreError: 167-0: atlas2-MDT0000-lwp-OST0360: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h8.ccs.ornl.gov kernel: [1918850.398315] Lustre: atlas2-MDT0000-lwp-OST039f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f4.ccs.ornl.gov kernel: [1918855.707005] Lustre: atlas2-MDT0000-lwp-OST026b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d6.ccs.ornl.gov kernel: [1918867.951453] LustreError: 167-0: atlas2-MDT0000-lwp-OST02ed: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3f7.ccs.ornl.gov kernel: [805184.656221] LustreError: 167-0: atlas2-MDT0000-lwp-OST026e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3d6.ccs.ornl.gov kernel: [1918867.952071] Lustre: atlas2-MDT0000-lwp-OST037d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a4.ccs.ornl.gov kernel: [1918855.094252] Lustre: atlas2-MDT0000-lwp-OST031b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h3.ccs.ornl.gov kernel: [1918849.166056] LustreError: 167-0: atlas2-MDT0000-lwp-OST030a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h3.ccs.ornl.gov kernel: [1918849.208158] LustreError: 167-0: atlas2-MDT0000-lwp-OST039a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3i7.ccs.ornl.gov kernel: [1918850.730360] Lustre: atlas2-MDT0000-lwp-OST0286: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i7.ccs.ornl.gov kernel: [1918850.768513] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss4e2.ccs.ornl.gov kernel: [1918867.410789] LustreError: 167-0: atlas2-MDT0000-lwp-OST0339: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h7.ccs.ornl.gov kernel: [1918860.306302] LustreError: 167-0: atlas2-MDT0000-lwp-OST027e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h7.ccs.ornl.gov kernel: [1918860.348186] LustreError: 167-0: atlas2-MDT0000-lwp-OST015e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c1.ccs.ornl.gov kernel: [1918859.098383] LustreError: 167-0: atlas2-MDT0000-lwp-OST0298: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f2.ccs.ornl.gov kernel: [1918857.159073] Lustre: atlas2-MDT0000-lwp-OST0341: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4f2.ccs.ornl.gov kernel: [1918857.197160] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4h8.ccs.ornl.gov kernel: [1918844.518568] LustreError: 167-0: atlas2-MDT0000-lwp-OST0237: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h8.ccs.ornl.gov kernel: [1918844.560258] LustreError: 167-0: atlas2-MDT0000-lwp-OST0357: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4b6.ccs.ornl.gov kernel: [1918851.198941] LustreError: 167-0: atlas2-MDT0000-lwp-OST0175: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4b6.ccs.ornl.gov kernel: [1918851.235113] LustreError: 167-0: atlas2-MDT0000-lwp-OST0205: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c4.ccs.ornl.gov kernel: [1918865.483157] Lustre: atlas2-MDT0000-lwp-OST032b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4c4.ccs.ornl.gov kernel: [1918865.512865] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4a5.ccs.ornl.gov kernel: [1918872.551074] Lustre: atlas2-MDT0000-lwp-OST016c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4d3.ccs.ornl.gov kernel: [1918850.059339] Lustre: atlas2-MDT0000-lwp-OST0212: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4d3.ccs.ornl.gov kernel: [1918850.097611] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss4c5.ccs.ornl.gov kernel: [1918849.342665] Lustre: atlas2-MDT0000-lwp-OST032c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4c5.ccs.ornl.gov kernel: [1918849.372476] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4b1.ccs.ornl.gov kernel: [1918858.102183] LustreError: 167-0: atlas2-MDT0000-lwp-OST0200: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4b1.ccs.ornl.gov kernel: [1918858.138666] LustreError: 167-0: atlas2-MDT0000-lwp-OST0320: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e1.ccs.ornl.gov kernel: [1918856.312359] Lustre: atlas2-MDT0000-lwp-OST00b0: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4c8.ccs.ornl.gov kernel: [1918845.413768] LustreError: 167-0: atlas2-MDT0000-lwp-OST029f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c8.ccs.ornl.gov kernel: [1918845.414417] Lustre: atlas2-MDT0000-lwp-OST020f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g7.ccs.ornl.gov kernel: [1917451.160315] LustreError: 167-0: atlas2-MDT0000-lwp-OST0276: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3g7.ccs.ornl.gov kernel: [1917451.202387] LustreError: 167-0: atlas2-MDT0000-lwp-OST0306: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h4.ccs.ornl.gov kernel: [1918873.038762] Lustre: atlas2-MDT0000-lwp-OST03e3: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4h4.ccs.ornl.gov kernel: [1918873.077034] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4c3.ccs.ornl.gov kernel: [1918850.378484] Lustre: atlas2-MDT0000-lwp-OST032a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4c3.ccs.ornl.gov kernel: [1918850.416872] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3b3.ccs.ornl.gov kernel: [1918846.560146] LustreError: 167-0: atlas2-MDT0000-lwp-OST036a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3b3.ccs.ornl.gov kernel: [1918846.602254] LustreError: 167-0: atlas2-MDT0000-lwp-OST009a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e7.ccs.ornl.gov kernel: [1918850.962421] LustreError: 167-0: atlas2-MDT0000-lwp-OST02ae: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e7.ccs.ornl.gov kernel: [1918850.963047] Lustre: atlas2-MDT0000-lwp-OST033e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f5.ccs.ornl.gov kernel: [805545.451554] Lustre: atlas2-MDT0000-lwp-OST014c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a2.ccs.ornl.gov kernel: [1918868.217569] LustreError: 167-0: atlas2-MDT0000-lwp-OST0319: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4a2.ccs.ornl.gov kernel: [1918868.218177] Lustre: atlas2-MDT0000-lwp-OST0289: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f4.ccs.ornl.gov kernel: [1918855.786943] Lustre: atlas2-MDT0000-lwp-OST02fb: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f4.ccs.ornl.gov kernel: [1918855.816770] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3d6.ccs.ornl.gov kernel: [1918868.031681] Lustre: atlas2-MDT0000-lwp-OST025d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d6.ccs.ornl.gov kernel: [1918868.061754] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3f8.ccs.ornl.gov kernel: [805555.966799] Lustre: atlas2-MDT0000-lwp-OST026f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g8.ccs.ornl.gov kernel: [1917451.663481] Lustre: atlas2-MDT0000-lwp-OST01e7: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a4.ccs.ornl.gov kernel: [1918855.173968] Lustre: atlas2-MDT0000-lwp-OST028b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g8.ccs.ornl.gov kernel: [1917451.701755] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4a4.ccs.ornl.gov kernel: [1918855.203847] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3h8.ccs.ornl.gov kernel: [1918850.478349] Lustre: atlas2-MDT0000-lwp-OST030f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h8.ccs.ornl.gov kernel: [1918850.516601] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4c2.ccs.ornl.gov kernel: [1918857.511261] LustreError: 167-0: atlas2-MDT0000-lwp-OST0329: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c2.ccs.ornl.gov kernel: [1918857.511876] Lustre: atlas2-MDT0000-lwp-OST03b9: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a5.ccs.ornl.gov kernel: [1918872.630761] Lustre: atlas2-MDT0000-lwp-OST01fc: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a5.ccs.ornl.gov kernel: [1918872.660615] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3g5.ccs.ornl.gov kernel: [1917455.176390] LustreError: 167-0: atlas2-MDT0000-lwp-OST0304: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3g5.ccs.ornl.gov kernel: [1917455.177054] Lustre: atlas2-MDT0000-lwp-OST0274: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g3.ccs.ornl.gov kernel: [1918864.789408] LustreError: 167-0: atlas2-MDT0000-lwp-OST034a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g3.ccs.ornl.gov kernel: [1918864.829566] LustreError: 167-0: atlas2-MDT0000-lwp-OST02ba: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f5.ccs.ornl.gov kernel: [1918851.565469] LustreError: 167-0: atlas2-MDT0000-lwp-OST02b4: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f5.ccs.ornl.gov kernel: [1918851.600163] LustreError: 167-0: atlas2-MDT0000-lwp-OST0344: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3a5.ccs.ornl.gov kernel: [1918847.160384] LustreError: 167-0: atlas2-MDT0000-lwp-OST01b4: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3a5.ccs.ornl.gov kernel: [1918847.198891] LustreError: 167-0: atlas2-MDT0000-lwp-OST0244: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3a6.ccs.ornl.gov kernel: [1918856.271828] LustreError: 167-0: atlas2-MDT0000-lwp-OST0245: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3a6.ccs.ornl.gov kernel: [1918856.307018] LustreError: 167-0: atlas2-MDT0000-lwp-OST02d5: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e7.ccs.ornl.gov kernel: [1918851.042875] Lustre: atlas2-MDT0000-lwp-OST02ae: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4e7.ccs.ornl.gov kernel: [1918851.072696] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss4a2.ccs.ornl.gov kernel: [1918868.298145] Lustre: atlas2-MDT0000-lwp-OST0319: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3e1.ccs.ornl.gov kernel: [1918856.392682] Lustre: atlas2-MDT0000-lwp-OST0140: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3e1.ccs.ornl.gov kernel: [1918856.431172] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4c8.ccs.ornl.gov kernel: [1918845.494673] Lustre: atlas2-MDT0000-lwp-OST029f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f8.ccs.ornl.gov kernel: [805556.046639] Lustre: atlas2-MDT0000-lwp-OST02ff: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f8.ccs.ornl.gov kernel: [805556.076527] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4e1.ccs.ornl.gov kernel: [1918851.457373] Lustre: atlas2-MDT0000-lwp-OST02a8: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f5.ccs.ornl.gov kernel: [805545.531620] Lustre: atlas2-MDT0000-lwp-OST00bc: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f5.ccs.ornl.gov kernel: [805545.569708] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss4c8.ccs.ornl.gov kernel: [1918845.532991] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4d4.ccs.ornl.gov kernel: [1918873.984204] Lustre: atlas2-MDT0000-lwp-OST03c3: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g8.ccs.ornl.gov kernel: [1918852.702950] LustreError: 167-0: atlas2-MDT0000-lwp-OST02bf: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g8.ccs.ornl.gov kernel: [1918852.703586] Lustre: atlas2-MDT0000-lwp-OST034f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3a1.ccs.ornl.gov kernel: [1918848.429552] Lustre: atlas2-MDT0000-lwp-OST0090: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a2.ccs.ornl.gov kernel: [1918868.327779] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4c2.ccs.ornl.gov kernel: [1918857.592145] Lustre: atlas2-MDT0000-lwp-OST0299: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4c2.ccs.ornl.gov kernel: [1918857.630287] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3f7.ccs.ornl.gov kernel: [805184.693962] LustreError: 167-0: atlas2-MDT0000-lwp-OST014e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3f7.ccs.ornl.gov kernel: [805184.694568] Lustre: atlas2-MDT0000-lwp-OST026e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h3.ccs.ornl.gov kernel: [1918849.208799] Lustre: atlas2-MDT0000-lwp-OST030a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4e2.ccs.ornl.gov kernel: [1918867.450599] LustreError: 167-0: atlas2-MDT0000-lwp-OST02a9: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e2.ccs.ornl.gov kernel: [1918867.451215] Lustre: atlas2-MDT0000-lwp-OST0339: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g5.ccs.ornl.gov kernel: [1917455.257011] Lustre: atlas2-MDT0000-lwp-OST0304: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h7.ccs.ornl.gov kernel: [1918860.348814] Lustre: atlas2-MDT0000-lwp-OST027e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g5.ccs.ornl.gov kernel: [1917455.295093] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4c1.ccs.ornl.gov kernel: [1918859.133088] LustreError: 167-0: atlas2-MDT0000-lwp-OST03b8: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c1.ccs.ornl.gov kernel: [1918859.133697] Lustre: atlas2-MDT0000-lwp-OST0298: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4h8.ccs.ornl.gov kernel: [1918844.560869] Lustre: atlas2-MDT0000-lwp-OST0237: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b6.ccs.ornl.gov kernel: [1918851.235749] Lustre: atlas2-MDT0000-lwp-OST0175: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4f6.ccs.ornl.gov kernel: [1918848.444331] LustreError: 167-0: atlas2-MDT0000-lwp-OST0195: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3b5.ccs.ornl.gov kernel: [1918866.701474] LustreError: 167-0: atlas2-MDT0000-lwp-OST036c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3i4.ccs.ornl.gov kernel: [1918859.548043] LustreError: 167-0: atlas2-MDT0000-lwp-OST00d3: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3i4.ccs.ornl.gov kernel: [1918859.590242] LustreError: 167-0: atlas2-MDT0000-lwp-OST03a3: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4b1.ccs.ornl.gov kernel: [1918858.139287] Lustre: atlas2-MDT0000-lwp-OST0200: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g7.ccs.ornl.gov kernel: [1917451.203040] Lustre: atlas2-MDT0000-lwp-OST0276: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4e1.ccs.ornl.gov kernel: [1918851.537199] Lustre: atlas2-MDT0000-lwp-OST0338: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g7.ccs.ornl.gov kernel: [1918876.015783] LustreError: 167-0: atlas2-MDT0000-lwp-OST02be: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g7.ccs.ornl.gov kernel: [1918876.052495] LustreError: 167-0: atlas2-MDT0000-lwp-OST019e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g4.ccs.ornl.gov kernel: [1918868.152228] LustreError: 167-0: atlas2-MDT0000-lwp-OST022b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g4.ccs.ornl.gov kernel: [1918868.194108] LustreError: 167-0: atlas2-MDT0000-lwp-OST019b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3b3.ccs.ornl.gov kernel: [1918846.602871] Lustre: atlas2-MDT0000-lwp-OST036a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b7.ccs.ornl.gov kernel: [1918847.746217] LustreError: 167-0: atlas2-MDT0000-lwp-OST0206: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f3.ccs.ornl.gov kernel: [1918848.401953] LustreError: 167-0: atlas2-MDT0000-lwp-OST0222: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4b7.ccs.ornl.gov kernel: [1918847.784648] LustreError: 167-0: atlas2-MDT0000-lwp-OST0176: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f3.ccs.ornl.gov kernel: [1918848.438929] LustreError: 167-0: atlas2-MDT0000-lwp-OST0342: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g6.ccs.ornl.gov kernel: [1918863.383194] LustreError: 167-0: atlas2-MDT0000-lwp-OST03dd: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g6.ccs.ornl.gov kernel: [1918863.425035] LustreError: 167-0: atlas2-MDT0000-lwp-OST034d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h2.ccs.ornl.gov kernel: [1918856.733997] LustreError: 167-0: atlas2-MDT0000-lwp-OST0309: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h4.ccs.ornl.gov kernel: [1918857.474681] LustreError: 167-0: atlas2-MDT0000-lwp-OST027b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d4.ccs.ornl.gov kernel: [1918874.064264] Lustre: atlas2-MDT0000-lwp-OST0333: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b8.ccs.ornl.gov kernel: [1918855.555727] LustreError: 167-0: atlas2-MDT0000-lwp-OST0207: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d4.ccs.ornl.gov kernel: [1918874.102493] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3h3.ccs.ornl.gov kernel: [1918849.289035] Lustre: atlas2-MDT0000-lwp-OST039a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h3.ccs.ornl.gov kernel: [1918849.319068] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4g8.ccs.ornl.gov kernel: [1918852.783753] Lustre: atlas2-MDT0000-lwp-OST02bf: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4e1.ccs.ornl.gov kernel: [1918851.575468] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4g8.ccs.ornl.gov kernel: [1918852.821747] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3a1.ccs.ornl.gov kernel: [1918848.509706] Lustre: atlas2-MDT0000-lwp-OST0360: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h7.ccs.ornl.gov kernel: [1918860.429120] Lustre: atlas2-MDT0000-lwp-OST015e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3a1.ccs.ornl.gov kernel: [1918848.548187] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3h7.ccs.ornl.gov kernel: [1918860.458970] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4c1.ccs.ornl.gov kernel: [1918859.213588] Lustre: atlas2-MDT0000-lwp-OST03b8: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4c1.ccs.ornl.gov kernel: [1918859.243415] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3f7.ccs.ornl.gov kernel: [805184.774718] Lustre: atlas2-MDT0000-lwp-OST014e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f7.ccs.ornl.gov kernel: [805184.812936] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3e7.ccs.ornl.gov kernel: [1918856.099643] LustreError: 167-0: atlas2-MDT0000-lwp-OST0266: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e7.ccs.ornl.gov kernel: [1918856.136583] LustreError: 167-0: atlas2-MDT0000-lwp-OST02f6: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e2.ccs.ornl.gov kernel: [1918867.531331] Lustre: atlas2-MDT0000-lwp-OST0219: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4e2.ccs.ornl.gov kernel: [1918867.569528] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss3g1.ccs.ornl.gov kernel: [1917443.657574] LustreError: 167-0: atlas2-MDT0000-lwp-OST0390: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3a2.ccs.ornl.gov kernel: [1918853.243537] LustreError: 167-0: atlas2-MDT0000-lwp-OST01b1: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3a2.ccs.ornl.gov kernel: [1918853.280234] LustreError: 167-0: atlas2-MDT0000-lwp-OST0121: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h8.ccs.ornl.gov kernel: [1918844.641233] Lustre: atlas2-MDT0000-lwp-OST0357: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4h8.ccs.ornl.gov kernel: [1918844.679576] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4b6.ccs.ornl.gov kernel: [1918851.315935] Lustre: atlas2-MDT0000-lwp-OST0205: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b6.ccs.ornl.gov kernel: [1918851.354029] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4b1.ccs.ornl.gov kernel: [1918858.219373] Lustre: atlas2-MDT0000-lwp-OST0320: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b1.ccs.ornl.gov kernel: [1918858.249327] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3g2.ccs.ornl.gov kernel: [1917450.346949] LustreError: 167-0: atlas2-MDT0000-lwp-OST0271: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3g7.ccs.ornl.gov kernel: [1917451.282948] Lustre: atlas2-MDT0000-lwp-OST0396: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g7.ccs.ornl.gov kernel: [1917451.312764] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4g3.ccs.ornl.gov kernel: [1918864.830185] Lustre: atlas2-MDT0000-lwp-OST034a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4f5.ccs.ornl.gov kernel: [1918851.600787] Lustre: atlas2-MDT0000-lwp-OST02b4: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3a5.ccs.ornl.gov kernel: [1918847.199532] Lustre: atlas2-MDT0000-lwp-OST01b4: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3a6.ccs.ornl.gov kernel: [1918856.307654] Lustre: atlas2-MDT0000-lwp-OST0245: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3b3.ccs.ornl.gov kernel: [1918846.682919] Lustre: atlas2-MDT0000-lwp-OST009a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3b3.ccs.ornl.gov kernel: [1918846.713044] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3h1.ccs.ornl.gov kernel: [1918842.411854] LustreError: 167-0: atlas2-MDT0000-lwp-OST0278: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h1.ccs.ornl.gov kernel: [1918842.446147] LustreError: 167-0: atlas2-MDT0000-lwp-OST0308: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d5.ccs.ornl.gov kernel: [1918866.257204] LustreError: 167-0: atlas2-MDT0000-lwp-OST03c4: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e8.ccs.ornl.gov kernel: [1918866.699300] LustreError: 167-0: atlas2-MDT0000-lwp-OST00ff: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e8.ccs.ornl.gov kernel: [1918866.739774] LustreError: 167-0: atlas2-MDT0000-lwp-OST018f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e3.ccs.ornl.gov kernel: [1918859.552604] LustreError: 167-0: atlas2-MDT0000-lwp-OST02aa: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e3.ccs.ornl.gov kernel: [1918859.590767] LustreError: 167-0: atlas2-MDT0000-lwp-OST033a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f1.ccs.ornl.gov kernel: [1918857.914843] LustreError: 167-0: atlas2-MDT0000-lwp-OST0100: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f1.ccs.ornl.gov kernel: [1918857.957115] LustreError: 167-0: atlas2-MDT0000-lwp-OST0190: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i8.ccs.ornl.gov kernel: [1918867.446171] LustreError: 167-0: atlas2-MDT0000-lwp-OST035f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i5.ccs.ornl.gov kernel: [1918861.706175] LustreError: 167-0: atlas2-MDT0000-lwp-OST011c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3i8.ccs.ornl.gov kernel: [1918867.660764] LustreError: 167-0: atlas2-MDT0000-lwp-OST0167: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3i8.ccs.ornl.gov kernel: [1918867.697476] LustreError: 167-0: atlas2-MDT0000-lwp-OST01f7: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3d8.ccs.ornl.gov kernel: [1918860.322270] LustreError: 167-0: atlas2-MDT0000-lwp-OST02ef: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3d8.ccs.ornl.gov kernel: [1918860.357284] LustreError: 167-0: atlas2-MDT0000-lwp-OST025f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3a6.ccs.ornl.gov kernel: [1918856.387552] Lustre: atlas2-MDT0000-lwp-OST02d5: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3a6.ccs.ornl.gov kernel: [1918856.417679] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4f6.ccs.ornl.gov kernel: [1918848.480234] LustreError: 167-0: atlas2-MDT0000-lwp-OST02b5: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3f2.ccs.ornl.gov kernel: [1918848.883692] LustreError: 167-0: atlas2-MDT0000-lwp-OST0269: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f6.ccs.ornl.gov kernel: [1918848.480823] Lustre: atlas2-MDT0000-lwp-OST0195: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3b5.ccs.ornl.gov kernel: [1918866.739561] LustreError: 167-0: atlas2-MDT0000-lwp-OST02dc: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3b5.ccs.ornl.gov kernel: [1918866.740153] Lustre: atlas2-MDT0000-lwp-OST036c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g3.ccs.ornl.gov kernel: [1918864.910492] Lustre: atlas2-MDT0000-lwp-OST02ba: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4f5.ccs.ornl.gov kernel: [1918851.680812] Lustre: atlas2-MDT0000-lwp-OST0344: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g3.ccs.ornl.gov kernel: [1918864.948674] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4f5.ccs.ornl.gov kernel: [1918851.719059] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3a5.ccs.ornl.gov kernel: [1918847.279529] Lustre: atlas2-MDT0000-lwp-OST0244: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3a5.ccs.ornl.gov kernel: [1918847.317803] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss4h5.ccs.ornl.gov kernel: [1918856.267170] LustreError: 167-0: atlas2-MDT0000-lwp-OST01a4: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3i4.ccs.ornl.gov kernel: [1918859.590857] Lustre: atlas2-MDT0000-lwp-OST00d3: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3a4.ccs.ornl.gov kernel: [1918843.731197] LustreError: 167-0: atlas2-MDT0000-lwp-OST0243: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3a4.ccs.ornl.gov kernel: [1918843.768678] LustreError: 167-0: atlas2-MDT0000-lwp-OST02d3: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c7.ccs.ornl.gov kernel: [1918853.634818] LustreError: 167-0: atlas2-MDT0000-lwp-OST029e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c7.ccs.ornl.gov kernel: [1918853.673197] LustreError: 167-0: atlas2-MDT0000-lwp-OST017e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4b5.ccs.ornl.gov kernel: [1918871.357615] LustreError: 167-0: atlas2-MDT0000-lwp-OST03b4: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4b5.ccs.ornl.gov kernel: [1918871.395741] LustreError: 167-0: atlas2-MDT0000-lwp-OST0294: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g4.ccs.ornl.gov kernel: [1918868.194720] Lustre: atlas2-MDT0000-lwp-OST022b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4f3.ccs.ornl.gov kernel: [1918848.439562] Lustre: atlas2-MDT0000-lwp-OST0222: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b7.ccs.ornl.gov kernel: [1918847.785276] Lustre: atlas2-MDT0000-lwp-OST0206: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g6.ccs.ornl.gov kernel: [1918863.425646] Lustre: atlas2-MDT0000-lwp-OST03dd: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d1.ccs.ornl.gov kernel: [1918846.432115] LustreError: 167-0: atlas2-MDT0000-lwp-OST02e8: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h2.ccs.ornl.gov kernel: [1918856.775912] LustreError: 167-0: atlas2-MDT0000-lwp-OST0399: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h2.ccs.ornl.gov kernel: [1918856.776539] Lustre: atlas2-MDT0000-lwp-OST0309: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4i6.ccs.ornl.gov kernel: [1918867.174709] LustreError: 167-0: atlas2-MDT0000-lwp-OST035d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i6.ccs.ornl.gov kernel: [1918867.213793] LustreError: 167-0: atlas2-MDT0000-lwp-OST023d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h4.ccs.ornl.gov kernel: [1918857.515301] LustreError: 167-0: atlas2-MDT0000-lwp-OST039b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h4.ccs.ornl.gov kernel: [1918857.515914] Lustre: atlas2-MDT0000-lwp-OST027b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b8.ccs.ornl.gov kernel: [1918855.591628] LustreError: 167-0: atlas2-MDT0000-lwp-OST0327: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g7.ccs.ornl.gov kernel: [1918876.053128] Lustre: atlas2-MDT0000-lwp-OST02be: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b8.ccs.ornl.gov kernel: [1918855.592242] Lustre: atlas2-MDT0000-lwp-OST0207: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4h1.ccs.ornl.gov kernel: [1918847.645262] LustreError: 167-0: atlas2-MDT0000-lwp-OST03e0: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h1.ccs.ornl.gov kernel: [1918847.684573] LustreError: 167-0: atlas2-MDT0000-lwp-OST0350: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e5.ccs.ornl.gov kernel: [1918854.874514] LustreError: 167-0: atlas2-MDT0000-lwp-OST033c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e8.ccs.ornl.gov kernel: [1918853.895654] LustreError: 167-0: atlas2-MDT0000-lwp-OST0267: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e5.ccs.ornl.gov kernel: [1918854.908092] LustreError: 167-0: atlas2-MDT0000-lwp-OST03cc: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e8.ccs.ornl.gov kernel: [1918853.932368] LustreError: 167-0: atlas2-MDT0000-lwp-OST02f7: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3b6.ccs.ornl.gov kernel: [1918852.005916] LustreError: 167-0: atlas2-MDT0000-lwp-OST012d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3b6.ccs.ornl.gov kernel: [1918852.043769] LustreError: 167-0: atlas2-MDT0000-lwp-OST01bd: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f4.ccs.ornl.gov kernel: [1918854.238066] LustreError: 167-0: atlas2-MDT0000-lwp-OST0193: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f4.ccs.ornl.gov kernel: [1918854.275758] LustreError: 167-0: atlas2-MDT0000-lwp-OST0223: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e7.ccs.ornl.gov kernel: [1918856.137214] Lustre: atlas2-MDT0000-lwp-OST0266: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3b2.ccs.ornl.gov kernel: [1918853.784370] LustreError: 167-0: atlas2-MDT0000-lwp-OST02d9: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3b5.ccs.ornl.gov kernel: [1918866.820232] Lustre: atlas2-MDT0000-lwp-OST02dc: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3b2.ccs.ornl.gov kernel: [1918853.821126] LustreError: 167-0: atlas2-MDT0000-lwp-OST0249: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3b5.ccs.ornl.gov kernel: [1918866.849561] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3g1.ccs.ornl.gov kernel: [1917443.696216] LustreError: 167-0: atlas2-MDT0000-lwp-OST0300: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3g1.ccs.ornl.gov kernel: [1917443.696815] Lustre: atlas2-MDT0000-lwp-OST0390: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a6.ccs.ornl.gov kernel: [1918855.863372] LustreError: 167-0: atlas2-MDT0000-lwp-OST00dd: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4a6.ccs.ornl.gov kernel: [1918855.897855] LustreError: 167-0: atlas2-MDT0000-lwp-OST016d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3a3.ccs.ornl.gov kernel: [1918851.014872] LustreError: 167-0: atlas2-MDT0000-lwp-OST0242: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3a3.ccs.ornl.gov kernel: [1918851.055755] LustreError: 167-0: atlas2-MDT0000-lwp-OST02d2: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f6.ccs.ornl.gov kernel: [1918848.560930] Lustre: atlas2-MDT0000-lwp-OST02b5: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g2.ccs.ornl.gov kernel: [1917450.388096] LustreError: 167-0: atlas2-MDT0000-lwp-OST0301: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f6.ccs.ornl.gov kernel: [1918848.599231] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3g2.ccs.ornl.gov kernel: [1917450.388762] Lustre: atlas2-MDT0000-lwp-OST0271: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g4.ccs.ornl.gov kernel: [1918868.274817] Lustre: atlas2-MDT0000-lwp-OST03db: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g4.ccs.ornl.gov kernel: [1918868.304817] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3e5.ccs.ornl.gov kernel: [1918855.037408] LustreError: 167-0: atlas2-MDT0000-lwp-OST02f4: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e5.ccs.ornl.gov kernel: [1918855.079009] LustreError: 167-0: atlas2-MDT0000-lwp-OST0264: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d6.ccs.ornl.gov kernel: [1918853.890815] LustreError: 167-0: atlas2-MDT0000-lwp-OST00f5: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d6.ccs.ornl.gov kernel: [1918853.925209] LustreError: 167-0: atlas2-MDT0000-lwp-OST0185: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3a2.ccs.ornl.gov kernel: [1918853.280870] Lustre: atlas2-MDT0000-lwp-OST01b1: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h1.ccs.ornl.gov kernel: [1918842.446797] Lustre: atlas2-MDT0000-lwp-OST0278: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g6.ccs.ornl.gov kernel: [1918863.505403] Lustre: atlas2-MDT0000-lwp-OST034d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g6.ccs.ornl.gov kernel: [1918863.535057] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3c5.ccs.ornl.gov kernel: [1918860.427225] LustreError: 167-0: atlas2-MDT0000-lwp-OST00a4: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3c5.ccs.ornl.gov kernel: [1918860.462745] LustreError: 167-0: atlas2-MDT0000-lwp-OST0374: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h2.ccs.ornl.gov kernel: [1918856.856398] Lustre: atlas2-MDT0000-lwp-OST0399: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h2.ccs.ornl.gov kernel: [1918856.886294] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3g4.ccs.ornl.gov kernel: [1917464.327811] LustreError: 167-0: atlas2-MDT0000-lwp-OST00c3: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3g4.ccs.ornl.gov kernel: [1917464.369044] LustreError: 167-0: atlas2-MDT0000-lwp-OST0153: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g7.ccs.ornl.gov kernel: [1918876.142666] Lustre: atlas2-MDT0000-lwp-OST019e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g7.ccs.ornl.gov kernel: [1918876.172590] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3c2.ccs.ornl.gov kernel: [1918855.857805] LustreError: 167-0: atlas2-MDT0000-lwp-OST01c1: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3c2.ccs.ornl.gov kernel: [1918855.899992] LustreError: 167-0: atlas2-MDT0000-lwp-OST0251: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g1.ccs.ornl.gov kernel: [1918850.880960] LustreError: 167-0: atlas2-MDT0000-lwp-OST0108: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g1.ccs.ornl.gov kernel: [1918850.922723] LustreError: 167-0: atlas2-MDT0000-lwp-OST0198: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f3.ccs.ornl.gov kernel: [1918848.519664] Lustre: atlas2-MDT0000-lwp-OST0342: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b7.ccs.ornl.gov kernel: [1918847.865483] Lustre: atlas2-MDT0000-lwp-OST00e6: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4f3.ccs.ornl.gov kernel: [1918848.557817] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4b7.ccs.ornl.gov kernel: [1918847.903531] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4d5.ccs.ornl.gov kernel: [1918866.292719] LustreError: 167-0: atlas2-MDT0000-lwp-OST0334: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d5.ccs.ornl.gov kernel: [1918866.293340] Lustre: atlas2-MDT0000-lwp-OST03c4: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f3.ccs.ornl.gov kernel: [1918867.356910] LustreError: 167-0: atlas2-MDT0000-lwp-OST02fa: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e8.ccs.ornl.gov kernel: [1918866.740420] Lustre: atlas2-MDT0000-lwp-OST00ff: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h4.ccs.ornl.gov kernel: [1918857.595933] Lustre: atlas2-MDT0000-lwp-OST039b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h4.ccs.ornl.gov kernel: [1918857.633991] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4e3.ccs.ornl.gov kernel: [1918859.591392] Lustre: atlas2-MDT0000-lwp-OST02aa: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g1.ccs.ornl.gov kernel: [1917443.776703] Lustre: atlas2-MDT0000-lwp-OST0300: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g1.ccs.ornl.gov kernel: [1917443.806777] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4f1.ccs.ornl.gov kernel: [1918857.957732] Lustre: atlas2-MDT0000-lwp-OST0100: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4i8.ccs.ornl.gov kernel: [1918867.488294] LustreError: 167-0: atlas2-MDT0000-lwp-OST03ef: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i8.ccs.ornl.gov kernel: [1918867.488927] Lustre: atlas2-MDT0000-lwp-OST035f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4i5.ccs.ornl.gov kernel: [1918861.742559] LustreError: 167-0: atlas2-MDT0000-lwp-OST03ec: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i5.ccs.ornl.gov kernel: [1918861.743193] Lustre: atlas2-MDT0000-lwp-OST011c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c7.ccs.ornl.gov kernel: [1918854.694898] LustreError: 167-0: atlas2-MDT0000-lwp-OST0256: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3i8.ccs.ornl.gov kernel: [1918867.698122] Lustre: atlas2-MDT0000-lwp-OST0167: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3e7.ccs.ornl.gov kernel: [1918856.217570] Lustre: atlas2-MDT0000-lwp-OST02f6: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3e7.ccs.ornl.gov kernel: [1918856.255856] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss3c6.ccs.ornl.gov kernel: [1918846.064672] LustreError: 167-0: atlas2-MDT0000-lwp-OST0255: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3i4.ccs.ornl.gov kernel: [1918859.671044] Lustre: atlas2-MDT0000-lwp-OST03a3: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i4.ccs.ornl.gov kernel: [1918859.671047] Lustre: atlas2-MDT0000-lwp-OST0313: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i4.ccs.ornl.gov kernel: [1918859.739668] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss3d8.ccs.ornl.gov kernel: [1918860.357921] Lustre: atlas2-MDT0000-lwp-OST02ef: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h1.ccs.ornl.gov kernel: [1918842.526927] Lustre: atlas2-MDT0000-lwp-OST0308: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3a2.ccs.ornl.gov kernel: [1918853.369454] Lustre: atlas2-MDT0000-lwp-OST0121: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h1.ccs.ornl.gov kernel: [1918842.556478] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3a2.ccs.ornl.gov kernel: [1918853.399191] Lustre: Skipped 2 previous similar messages Jul 17 17:54:57 atlas-oss3e4.ccs.ornl.gov kernel: [1918868.100457] LustreError: 167-0: atlas2-MDT0000-lwp-OST0143: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3g2.ccs.ornl.gov kernel: [1917450.468591] Lustre: atlas2-MDT0000-lwp-OST0391: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3e4.ccs.ornl.gov kernel: [1918868.141371] LustreError: 167-0: atlas2-MDT0000-lwp-OST01d3: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3g2.ccs.ornl.gov kernel: [1917450.506946] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4c6.ccs.ornl.gov kernel: [1918854.751208] LustreError: 167-0: atlas2-MDT0000-lwp-OST017d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c6.ccs.ornl.gov kernel: [1918854.786382] LustreError: 167-0: atlas2-MDT0000-lwp-OST020d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3f2.ccs.ornl.gov kernel: [1918848.919593] LustreError: 167-0: atlas2-MDT0000-lwp-OST00b9: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3f2.ccs.ornl.gov kernel: [1918848.920225] Lustre: atlas2-MDT0000-lwp-OST0269: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4h5.ccs.ornl.gov kernel: [1918856.308300] LustreError: 167-0: atlas2-MDT0000-lwp-OST0234: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h5.ccs.ornl.gov kernel: [1918856.308941] Lustre: atlas2-MDT0000-lwp-OST01a4: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d3.ccs.ornl.gov kernel: [1918860.671296] LustreError: 167-0: atlas2-MDT0000-lwp-OST037a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3d3.ccs.ornl.gov kernel: [1918860.713408] LustreError: 167-0: atlas2-MDT0000-lwp-OST00aa: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4a3.ccs.ornl.gov kernel: [1918869.823074] LustreError: 167-0: atlas2-MDT0000-lwp-OST031a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4a3.ccs.ornl.gov kernel: [1918869.865122] LustreError: 167-0: atlas2-MDT0000-lwp-OST028a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d5.ccs.ornl.gov kernel: [1918866.373358] Lustre: atlas2-MDT0000-lwp-OST0184: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c8.ccs.ornl.gov kernel: [1918858.890823] LustreError: 167-0: atlas2-MDT0000-lwp-OST0377: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d5.ccs.ornl.gov kernel: [1918866.403182] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3a4.ccs.ornl.gov kernel: [1918843.769321] Lustre: atlas2-MDT0000-lwp-OST0243: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a1.ccs.ornl.gov kernel: [1918851.787229] LustreError: 167-0: atlas2-MDT0000-lwp-OST0288: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4a1.ccs.ornl.gov kernel: [1918851.824216] LustreError: 167-0: atlas2-MDT0000-lwp-OST0318: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4a7.ccs.ornl.gov kernel: [1918865.520032] LustreError: 167-0: atlas2-MDT0000-lwp-OST01fe: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4a7.ccs.ornl.gov kernel: [1918865.560999] LustreError: 167-0: atlas2-MDT0000-lwp-OST031e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c7.ccs.ornl.gov kernel: [1918853.673848] Lustre: atlas2-MDT0000-lwp-OST029e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4e6.ccs.ornl.gov kernel: [1918869.628600] LustreError: 167-0: atlas2-MDT0000-lwp-OST03cd: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e3.ccs.ornl.gov kernel: [1918859.671701] Lustre: atlas2-MDT0000-lwp-OST033a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b8.ccs.ornl.gov kernel: [1918855.672396] Lustre: atlas2-MDT0000-lwp-OST0327: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4e6.ccs.ornl.gov kernel: [1918869.663328] LustreError: 167-0: atlas2-MDT0000-lwp-OST02ad: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e3.ccs.ornl.gov kernel: [1918859.701274] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4b8.ccs.ornl.gov kernel: [1918855.672399] Lustre: atlas2-MDT0000-lwp-OST03b7: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b8.ccs.ornl.gov kernel: [1918855.740982] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss4b5.ccs.ornl.gov kernel: [1918871.396335] Lustre: atlas2-MDT0000-lwp-OST03b4: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3e2.ccs.ornl.gov kernel: [1918853.233069] LustreError: 167-0: atlas2-MDT0000-lwp-OST0141: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e2.ccs.ornl.gov kernel: [1918853.274900] LustreError: 167-0: atlas2-MDT0000-lwp-OST01d1: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e8.ccs.ornl.gov kernel: [1918866.820870] Lustre: atlas2-MDT0000-lwp-OST018f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4e8.ccs.ornl.gov kernel: [1918866.858866] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss4f1.ccs.ornl.gov kernel: [1918858.037379] Lustre: atlas2-MDT0000-lwp-OST0220: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4f1.ccs.ornl.gov kernel: [1918858.067511] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4i8.ccs.ornl.gov kernel: [1918867.569072] Lustre: atlas2-MDT0000-lwp-OST03ef: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4i8.ccs.ornl.gov kernel: [1918867.598918] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3d1.ccs.ornl.gov kernel: [1918846.468006] LustreError: 167-0: atlas2-MDT0000-lwp-OST0378: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3d1.ccs.ornl.gov kernel: [1918846.468613] Lustre: atlas2-MDT0000-lwp-OST02e8: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4i5.ccs.ornl.gov kernel: [1918861.823221] Lustre: atlas2-MDT0000-lwp-OST03ec: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4i5.ccs.ornl.gov kernel: [1918861.852924] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4i6.ccs.ornl.gov kernel: [1918867.214427] Lustre: atlas2-MDT0000-lwp-OST035d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h6.ccs.ornl.gov kernel: [1918854.233073] LustreError: 167-0: atlas2-MDT0000-lwp-OST015d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h6.ccs.ornl.gov kernel: [1918854.271933] LustreError: 167-0: atlas2-MDT0000-lwp-OST01ed: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h1.ccs.ornl.gov kernel: [1918847.685249] Lustre: atlas2-MDT0000-lwp-OST03e0: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d2.ccs.ornl.gov kernel: [1918855.572430] LustreError: 167-0: atlas2-MDT0000-lwp-OST00a9: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4e5.ccs.ornl.gov kernel: [1918854.908735] Lustre: atlas2-MDT0000-lwp-OST033c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3e8.ccs.ornl.gov kernel: [1918853.932997] Lustre: atlas2-MDT0000-lwp-OST0267: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d2.ccs.ornl.gov kernel: [1918855.606378] LustreError: 167-0: atlas2-MDT0000-lwp-OST0139: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3b6.ccs.ornl.gov kernel: [1918852.044398] Lustre: atlas2-MDT0000-lwp-OST012d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d8.ccs.ornl.gov kernel: [1918860.437546] Lustre: atlas2-MDT0000-lwp-OST025f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d8.ccs.ornl.gov kernel: [1918860.467464] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss4f4.ccs.ornl.gov kernel: [1918854.276402] Lustre: atlas2-MDT0000-lwp-OST0193: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i8.ccs.ornl.gov kernel: [1918867.778477] Lustre: atlas2-MDT0000-lwp-OST0287: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i8.ccs.ornl.gov kernel: [1918867.816734] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3b2.ccs.ornl.gov kernel: [1918853.821736] Lustre: atlas2-MDT0000-lwp-OST02d9: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i6.ccs.ornl.gov kernel: [1918852.718063] LustreError: 167-0: atlas2-MDT0000-lwp-OST0315: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3i6.ccs.ornl.gov kernel: [1918852.756109] LustreError: 167-0: atlas2-MDT0000-lwp-OST0165: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3f2.ccs.ornl.gov kernel: [1918849.000274] Lustre: atlas2-MDT0000-lwp-OST00b9: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f2.ccs.ornl.gov kernel: [1918849.030264] Lustre: Skipped 2 previous similar messages Jul 17 17:54:57 atlas-oss4h5.ccs.ornl.gov kernel: [1918856.388503] Lustre: atlas2-MDT0000-lwp-OST0234: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4h5.ccs.ornl.gov kernel: [1918856.418466] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4a6.ccs.ornl.gov kernel: [1918855.898476] Lustre: atlas2-MDT0000-lwp-OST00dd: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3a3.ccs.ornl.gov kernel: [1918851.056426] Lustre: atlas2-MDT0000-lwp-OST0242: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c4.ccs.ornl.gov kernel: [1918849.151480] LustreError: 167-0: atlas2-MDT0000-lwp-OST02e3: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3g3.ccs.ornl.gov kernel: [1917451.267595] LustreError: 167-0: atlas2-MDT0000-lwp-OST0152: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3c4.ccs.ornl.gov kernel: [1918849.192811] LustreError: 167-0: atlas2-MDT0000-lwp-OST0373: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3g3.ccs.ornl.gov kernel: [1917451.304797] LustreError: 167-0: atlas2-MDT0000-lwp-OST0272: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d1.ccs.ornl.gov kernel: [1918849.899896] LustreError: 167-0: atlas2-MDT0000-lwp-OST03c0: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d1.ccs.ornl.gov kernel: [1918849.938157] LustreError: 167-0: atlas2-MDT0000-lwp-OST02a0: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3a7.ccs.ornl.gov kernel: [1918852.006311] LustreError: 167-0: atlas2-MDT0000-lwp-OST0246: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3a7.ccs.ornl.gov kernel: [1918852.039720] LustreError: 167-0: atlas2-MDT0000-lwp-OST02d6: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e5.ccs.ornl.gov kernel: [1918855.079650] Lustre: atlas2-MDT0000-lwp-OST02f4: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i3.ccs.ornl.gov kernel: [1918849.349067] LustreError: 167-0: atlas2-MDT0000-lwp-OST03a2: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3i3.ccs.ornl.gov kernel: [1918849.384971] LustreError: 167-0: atlas2-MDT0000-lwp-OST0312: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d6.ccs.ornl.gov kernel: [1918853.925870] Lustre: atlas2-MDT0000-lwp-OST00f5: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3a4.ccs.ornl.gov kernel: [1918843.849493] Lustre: atlas2-MDT0000-lwp-OST02d3: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3a4.ccs.ornl.gov kernel: [1918843.887600] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3c5.ccs.ornl.gov kernel: [1918860.463388] Lustre: atlas2-MDT0000-lwp-OST00a4: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4c7.ccs.ornl.gov kernel: [1918853.753987] Lustre: atlas2-MDT0000-lwp-OST017e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g4.ccs.ornl.gov kernel: [1917464.369688] Lustre: atlas2-MDT0000-lwp-OST00c3: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b5.ccs.ornl.gov kernel: [1918871.476561] Lustre: atlas2-MDT0000-lwp-OST0294: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b5.ccs.ornl.gov kernel: [1918871.514810] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3g6.ccs.ornl.gov kernel: [1917462.283293] LustreError: 167-0: atlas2-MDT0000-lwp-OST00c5: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3c2.ccs.ornl.gov kernel: [1918855.900632] Lustre: atlas2-MDT0000-lwp-OST01c1: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g6.ccs.ornl.gov kernel: [1917462.320413] LustreError: 167-0: atlas2-MDT0000-lwp-OST0155: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4g1.ccs.ornl.gov kernel: [1918850.923355] Lustre: atlas2-MDT0000-lwp-OST0108: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4e5.ccs.ornl.gov kernel: [1918854.988857] Lustre: atlas2-MDT0000-lwp-OST03cc: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4e5.ccs.ornl.gov kernel: [1918855.018488] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3e8.ccs.ornl.gov kernel: [1918854.013180] Lustre: atlas2-MDT0000-lwp-OST02f7: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3e8.ccs.ornl.gov kernel: [1918854.043022] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3d1.ccs.ornl.gov kernel: [1918846.548822] Lustre: atlas2-MDT0000-lwp-OST0378: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d1.ccs.ornl.gov kernel: [1918846.586909] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3f3.ccs.ornl.gov kernel: [1918867.396271] LustreError: 167-0: atlas2-MDT0000-lwp-OST01da: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3f3.ccs.ornl.gov kernel: [1918867.396935] Lustre: atlas2-MDT0000-lwp-OST02fa: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4i6.ccs.ornl.gov kernel: [1918867.294660] Lustre: atlas2-MDT0000-lwp-OST023d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4i6.ccs.ornl.gov kernel: [1918867.332975] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3e6.ccs.ornl.gov kernel: [1918851.464195] LustreError: 167-0: atlas2-MDT0000-lwp-OST0265: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e6.ccs.ornl.gov kernel: [1918851.498063] LustreError: 167-0: atlas2-MDT0000-lwp-OST02f5: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4c7.ccs.ornl.gov kernel: [1918853.792200] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3b2.ccs.ornl.gov kernel: [1918853.901748] Lustre: atlas2-MDT0000-lwp-OST0249: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3b2.ccs.ornl.gov kernel: [1918853.931293] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4b4.ccs.ornl.gov kernel: [1918852.001874] LustreError: 167-0: atlas2-MDT0000-lwp-OST0323: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4b4.ccs.ornl.gov kernel: [1918852.035876] LustreError: 167-0: atlas2-MDT0000-lwp-OST03b3: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4h1.ccs.ornl.gov kernel: [1918847.765335] Lustre: atlas2-MDT0000-lwp-OST0350: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4h1.ccs.ornl.gov kernel: [1918847.803490] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4h3.ccs.ornl.gov kernel: [1918866.599052] LustreError: 167-0: atlas2-MDT0000-lwp-OST0352: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3b6.ccs.ornl.gov kernel: [1918852.124626] Lustre: atlas2-MDT0000-lwp-OST01bd: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3b6.ccs.ornl.gov kernel: [1918852.162866] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4h3.ccs.ornl.gov kernel: [1918866.640656] LustreError: 167-0: atlas2-MDT0000-lwp-OST0232: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3c3.ccs.ornl.gov kernel: [1918846.577264] LustreError: 167-0: atlas2-MDT0000-lwp-OST02e2: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3c3.ccs.ornl.gov kernel: [1918846.618034] LustreError: 167-0: atlas2-MDT0000-lwp-OST0252: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3d7.ccs.ornl.gov kernel: [1918854.767874] LustreError: 167-0: atlas2-MDT0000-lwp-OST01ce: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3d7.ccs.ornl.gov kernel: [1918854.807974] LustreError: 167-0: atlas2-MDT0000-lwp-OST025e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f4.ccs.ornl.gov kernel: [1918854.356380] Lustre: atlas2-MDT0000-lwp-OST0223: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4f4.ccs.ornl.gov kernel: [1918854.394642] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4a8.ccs.ornl.gov kernel: [1918859.955193] LustreError: 167-0: atlas2-MDT0000-lwp-OST028f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4a8.ccs.ornl.gov kernel: [1918859.994880] LustreError: 167-0: atlas2-MDT0000-lwp-OST031f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i1.ccs.ornl.gov kernel: [1918854.014512] LustreError: 167-0: atlas2-MDT0000-lwp-OST01a8: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4i1.ccs.ornl.gov kernel: [1918854.056980] LustreError: 167-0: atlas2-MDT0000-lwp-OST02c8: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3c7.ccs.ornl.gov kernel: [1918854.735807] LustreError: 167-0: atlas2-MDT0000-lwp-OST02e6: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3c7.ccs.ornl.gov kernel: [1918854.736412] Lustre: atlas2-MDT0000-lwp-OST0256: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c6.ccs.ornl.gov kernel: [1918846.106660] LustreError: 167-0: atlas2-MDT0000-lwp-OST02e5: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3c6.ccs.ornl.gov kernel: [1918846.107269] Lustre: atlas2-MDT0000-lwp-OST0255: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f1.ccs.ornl.gov kernel: [1918857.746063] LustreError: 167-0: atlas2-MDT0000-lwp-OST0268: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3f1.ccs.ornl.gov kernel: [1918857.787110] LustreError: 167-0: atlas2-MDT0000-lwp-OST0388: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4a6.ccs.ornl.gov kernel: [1918855.978835] Lustre: atlas2-MDT0000-lwp-OST016d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a6.ccs.ornl.gov kernel: [1918856.017020] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3a3.ccs.ornl.gov kernel: [1918851.136820] Lustre: atlas2-MDT0000-lwp-OST02d2: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4f8.ccs.ornl.gov kernel: [1918853.155613] LustreError: 167-0: atlas2-MDT0000-lwp-OST02b7: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4f8.ccs.ornl.gov kernel: [1918853.197910] LustreError: 167-0: atlas2-MDT0000-lwp-OST0347: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e5.ccs.ornl.gov kernel: [1918855.159739] Lustre: atlas2-MDT0000-lwp-OST0264: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3e5.ccs.ornl.gov kernel: [1918855.189750] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3c5.ccs.ornl.gov kernel: [1918860.543040] Lustre: atlas2-MDT0000-lwp-OST02e4: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3b7.ccs.ornl.gov kernel: [1918858.802502] LustreError: 167-0: atlas2-MDT0000-lwp-OST02de: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3c5.ccs.ornl.gov kernel: [1918860.573006] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3b7.ccs.ornl.gov kernel: [1918858.844309] LustreError: 167-0: atlas2-MDT0000-lwp-OST024e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3e4.ccs.ornl.gov kernel: [1918868.141958] Lustre: atlas2-MDT0000-lwp-OST0143: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4c6.ccs.ornl.gov kernel: [1918854.787002] Lustre: atlas2-MDT0000-lwp-OST017d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g4.ccs.ornl.gov kernel: [1917464.449469] Lustre: atlas2-MDT0000-lwp-OST0153: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g4.ccs.ornl.gov kernel: [1917464.479549] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss3b1.ccs.ornl.gov kernel: [1918860.583068] LustreError: 167-0: atlas2-MDT0000-lwp-OST0248: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3b1.ccs.ornl.gov kernel: [1918860.621682] LustreError: 167-0: atlas2-MDT0000-lwp-OST0368: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d6.ccs.ornl.gov kernel: [1918854.006030] Lustre: atlas2-MDT0000-lwp-OST0185: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d3.ccs.ornl.gov kernel: [1918860.714024] Lustre: atlas2-MDT0000-lwp-OST037a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4d6.ccs.ornl.gov kernel: [1918854.044304] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4g1.ccs.ornl.gov kernel: [1918851.003529] Lustre: atlas2-MDT0000-lwp-OST0198: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4g1.ccs.ornl.gov kernel: [1918851.033149] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4a3.ccs.ornl.gov kernel: [1918869.865733] Lustre: atlas2-MDT0000-lwp-OST031a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3a3.ccs.ornl.gov kernel: [1918851.174872] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3c8.ccs.ornl.gov kernel: [1918858.930614] LustreError: 167-0: atlas2-MDT0000-lwp-OST0257: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3c8.ccs.ornl.gov kernel: [1918858.931243] Lustre: atlas2-MDT0000-lwp-OST0377: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a1.ccs.ornl.gov kernel: [1918851.824848] Lustre: atlas2-MDT0000-lwp-OST0288: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i1.ccs.ornl.gov kernel: [1918854.076100] LustreError: 167-0: atlas2-MDT0000-lwp-OST0310: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3i1.ccs.ornl.gov kernel: [1918854.111581] LustreError: 167-0: atlas2-MDT0000-lwp-OST00d0: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4a7.ccs.ornl.gov kernel: [1918865.561644] Lustre: atlas2-MDT0000-lwp-OST01fe: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4e6.ccs.ornl.gov kernel: [1918869.663960] Lustre: atlas2-MDT0000-lwp-OST03cd: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c2.ccs.ornl.gov kernel: [1918855.980874] Lustre: atlas2-MDT0000-lwp-OST00a1: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c2.ccs.ornl.gov kernel: [1918856.018866] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3e2.ccs.ornl.gov kernel: [1918853.275529] Lustre: atlas2-MDT0000-lwp-OST0141: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f3.ccs.ornl.gov kernel: [1918867.477053] Lustre: atlas2-MDT0000-lwp-OST014a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f3.ccs.ornl.gov kernel: [1918867.515127] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3h6.ccs.ornl.gov kernel: [1918854.272544] Lustre: atlas2-MDT0000-lwp-OST015d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c6.ccs.ornl.gov kernel: [1918846.187100] Lustre: atlas2-MDT0000-lwp-OST02e5: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c6.ccs.ornl.gov kernel: [1918846.216991] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3d2.ccs.ornl.gov kernel: [1918855.606986] Lustre: atlas2-MDT0000-lwp-OST00a9: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4d8.ccs.ornl.gov kernel: [1918867.010351] LustreError: 167-0: atlas2-MDT0000-lwp-OST00f7: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss4d8.ccs.ornl.gov kernel: [1918867.052272] LustreError: 167-0: atlas2-MDT0000-lwp-OST0187: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3c7.ccs.ornl.gov kernel: [1918854.816774] Lustre: atlas2-MDT0000-lwp-OST02e6: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c7.ccs.ornl.gov kernel: [1918854.854792] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss3i6.ccs.ornl.gov kernel: [1918852.756747] Lustre: atlas2-MDT0000-lwp-OST0315: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d3.ccs.ornl.gov kernel: [1918860.793998] Lustre: atlas2-MDT0000-lwp-OST00aa: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d3.ccs.ornl.gov kernel: [1918860.823997] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4a3.ccs.ornl.gov kernel: [1918869.946030] Lustre: atlas2-MDT0000-lwp-OST028a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a3.ccs.ornl.gov kernel: [1918869.975908] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3e4.ccs.ornl.gov kernel: [1918868.222147] Lustre: atlas2-MDT0000-lwp-OST00b3: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3e4.ccs.ornl.gov kernel: [1918868.260219] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3c4.ccs.ornl.gov kernel: [1918849.193430] Lustre: atlas2-MDT0000-lwp-OST02e3: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4c6.ccs.ornl.gov kernel: [1918854.867198] Lustre: atlas2-MDT0000-lwp-OST020d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4c6.ccs.ornl.gov kernel: [1918854.905446] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3g3.ccs.ornl.gov kernel: [1917451.305432] Lustre: atlas2-MDT0000-lwp-OST0152: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a7.ccs.ornl.gov kernel: [1918865.641575] Lustre: atlas2-MDT0000-lwp-OST031e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a7.ccs.ornl.gov kernel: [1918865.671481] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4e6.ccs.ornl.gov kernel: [1918869.743844] Lustre: atlas2-MDT0000-lwp-OST02ad: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4e6.ccs.ornl.gov kernel: [1918869.773501] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4d1.ccs.ornl.gov kernel: [1918849.938780] Lustre: atlas2-MDT0000-lwp-OST03c0: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3a7.ccs.ornl.gov kernel: [1918852.040363] Lustre: atlas2-MDT0000-lwp-OST0246: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3e2.ccs.ornl.gov kernel: [1918853.355532] Lustre: atlas2-MDT0000-lwp-OST01d1: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i3.ccs.ornl.gov kernel: [1918849.385605] Lustre: atlas2-MDT0000-lwp-OST03a2: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3e2.ccs.ornl.gov kernel: [1918853.385453] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss3c8.ccs.ornl.gov kernel: [1918859.010955] Lustre: atlas2-MDT0000-lwp-OST0257: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c8.ccs.ornl.gov kernel: [1918859.049476] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4a1.ccs.ornl.gov kernel: [1918851.905084] Lustre: atlas2-MDT0000-lwp-OST0318: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3b8.ccs.ornl.gov kernel: [1918868.137394] LustreError: 167-0: atlas2-MDT0000-lwp-OST024f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3b8.ccs.ornl.gov kernel: [1918868.179409] LustreError: 167-0: atlas2-MDT0000-lwp-OST009f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3h6.ccs.ornl.gov kernel: [1918854.352517] Lustre: atlas2-MDT0000-lwp-OST01ed: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3h6.ccs.ornl.gov kernel: [1918854.382270] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3g6.ccs.ornl.gov kernel: [1917462.321032] Lustre: atlas2-MDT0000-lwp-OST00c5: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d2.ccs.ornl.gov kernel: [1918855.687026] Lustre: atlas2-MDT0000-lwp-OST0139: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d2.ccs.ornl.gov kernel: [1918855.716625] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss4a1.ccs.ornl.gov kernel: [1918851.943237] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3e6.ccs.ornl.gov kernel: [1918851.498696] Lustre: atlas2-MDT0000-lwp-OST0265: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i6.ccs.ornl.gov kernel: [1918852.836814] Lustre: atlas2-MDT0000-lwp-OST0165: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i6.ccs.ornl.gov kernel: [1918852.866506] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4b4.ccs.ornl.gov kernel: [1918852.036494] Lustre: atlas2-MDT0000-lwp-OST0323: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4h3.ccs.ornl.gov kernel: [1918866.641281] Lustre: atlas2-MDT0000-lwp-OST0352: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c3.ccs.ornl.gov kernel: [1918846.618657] Lustre: atlas2-MDT0000-lwp-OST02e2: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d7.ccs.ornl.gov kernel: [1918854.808607] Lustre: atlas2-MDT0000-lwp-OST01ce: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a8.ccs.ornl.gov kernel: [1918859.995533] Lustre: atlas2-MDT0000-lwp-OST028f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4i1.ccs.ornl.gov kernel: [1918854.057622] Lustre: atlas2-MDT0000-lwp-OST01a8: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d5.ccs.ornl.gov kernel: [1918857.477879] LustreError: 167-0: atlas2-MDT0000-lwp-OST01cc: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3d5.ccs.ornl.gov kernel: [1918857.511967] LustreError: 167-0: atlas2-MDT0000-lwp-OST013c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3c4.ccs.ornl.gov kernel: [1918849.273581] Lustre: atlas2-MDT0000-lwp-OST0373: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c4.ccs.ornl.gov kernel: [1918849.303273] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4f8.ccs.ornl.gov kernel: [1918853.198534] Lustre: atlas2-MDT0000-lwp-OST02b7: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g3.ccs.ornl.gov kernel: [1917451.385912] Lustre: atlas2-MDT0000-lwp-OST0272: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g3.ccs.ornl.gov kernel: [1917451.424090] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss3b7.ccs.ornl.gov kernel: [1918858.844940] Lustre: atlas2-MDT0000-lwp-OST02de: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f1.ccs.ornl.gov kernel: [1918857.787764] Lustre: atlas2-MDT0000-lwp-OST0268: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3a7.ccs.ornl.gov kernel: [1918852.120477] Lustre: atlas2-MDT0000-lwp-OST0366: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4d1.ccs.ornl.gov kernel: [1918850.018993] Lustre: atlas2-MDT0000-lwp-OST02a0: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3a7.ccs.ornl.gov kernel: [1918852.158610] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4d1.ccs.ornl.gov kernel: [1918850.057436] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3b1.ccs.ornl.gov kernel: [1918860.622336] Lustre: atlas2-MDT0000-lwp-OST0248: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i3.ccs.ornl.gov kernel: [1918849.465613] Lustre: atlas2-MDT0000-lwp-OST0282: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i3.ccs.ornl.gov kernel: [1918849.503715] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3i1.ccs.ornl.gov kernel: [1918854.112222] Lustre: atlas2-MDT0000-lwp-OST0310: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g6.ccs.ornl.gov kernel: [1917462.401274] Lustre: atlas2-MDT0000-lwp-OST0155: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3g6.ccs.ornl.gov kernel: [1917462.439628] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4h3.ccs.ornl.gov kernel: [1918866.721563] Lustre: atlas2-MDT0000-lwp-OST02c2: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4h3.ccs.ornl.gov kernel: [1918866.751070] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3e6.ccs.ornl.gov kernel: [1918851.578932] Lustre: atlas2-MDT0000-lwp-OST02f5: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3e6.ccs.ornl.gov kernel: [1918851.617059] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4b4.ccs.ornl.gov kernel: [1918852.116659] Lustre: atlas2-MDT0000-lwp-OST03b3: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4b4.ccs.ornl.gov kernel: [1918852.154942] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3c3.ccs.ornl.gov kernel: [1918846.698794] Lustre: atlas2-MDT0000-lwp-OST0252: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3c3.ccs.ornl.gov kernel: [1918846.736938] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss3d7.ccs.ornl.gov kernel: [1918854.888934] Lustre: atlas2-MDT0000-lwp-OST025e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d7.ccs.ornl.gov kernel: [1918854.926872] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4f8.ccs.ornl.gov kernel: [1918853.278627] Lustre: atlas2-MDT0000-lwp-OST0347: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4f8.ccs.ornl.gov kernel: [1918853.308457] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4d8.ccs.ornl.gov kernel: [1918867.052882] Lustre: atlas2-MDT0000-lwp-OST00f7: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a8.ccs.ornl.gov kernel: [1918860.075879] Lustre: atlas2-MDT0000-lwp-OST00df: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4a8.ccs.ornl.gov kernel: [1918860.114145] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss4i1.ccs.ornl.gov kernel: [1918854.137490] Lustre: atlas2-MDT0000-lwp-OST02c8: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4i1.ccs.ornl.gov kernel: [1918854.175722] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3b7.ccs.ornl.gov kernel: [1918858.925209] Lustre: atlas2-MDT0000-lwp-OST024e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3b7.ccs.ornl.gov kernel: [1918858.955042] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss3f1.ccs.ornl.gov kernel: [1918857.877179] Lustre: atlas2-MDT0000-lwp-OST0388: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3f1.ccs.ornl.gov kernel: [1918857.907055] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3b1.ccs.ornl.gov kernel: [1918860.702223] Lustre: atlas2-MDT0000-lwp-OST0098: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3b1.ccs.ornl.gov kernel: [1918860.732047] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3b8.ccs.ornl.gov kernel: [1918868.180023] Lustre: atlas2-MDT0000-lwp-OST024f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i1.ccs.ornl.gov kernel: [1918854.192369] Lustre: atlas2-MDT0000-lwp-OST00d0: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3i1.ccs.ornl.gov kernel: [1918854.230615] Lustre: Skipped 3 previous similar messages Jul 17 17:54:57 atlas-oss4d8.ccs.ornl.gov kernel: [1918867.133123] Lustre: atlas2-MDT0000-lwp-OST0187: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss4d8.ccs.ornl.gov kernel: [1918867.163194] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3d5.ccs.ornl.gov kernel: [1918857.512601] Lustre: atlas2-MDT0000-lwp-OST01cc: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3b4.ccs.ornl.gov kernel: [1918865.657155] LustreError: 167-0: atlas2-MDT0000-lwp-OST012b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3b4.ccs.ornl.gov kernel: [1918865.692052] LustreError: 167-0: atlas2-MDT0000-lwp-OST01bb: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3b8.ccs.ornl.gov kernel: [1918868.251493] Lustre: atlas2-MDT0000-lwp-OST01bf: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3b8.ccs.ornl.gov kernel: [1918868.290001] Lustre: Skipped 2 previous similar messages Jul 17 17:54:57 atlas-oss3d5.ccs.ornl.gov kernel: [1918857.592613] Lustre: atlas2-MDT0000-lwp-OST013c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3d5.ccs.ornl.gov kernel: [1918857.631004] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3b4.ccs.ornl.gov kernel: [1918865.692667] Lustre: atlas2-MDT0000-lwp-OST012b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3b4.ccs.ornl.gov kernel: [1918865.772692] Lustre: atlas2-MDT0000-lwp-OST01bb: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:57 atlas-oss3b4.ccs.ornl.gov kernel: [1918865.811105] Lustre: Skipped 4 previous similar messages Jul 17 17:54:57 atlas-oss3i2.ccs.ornl.gov kernel: [1918860.193732] LustreError: 167-0: atlas2-MDT0000-lwp-OST0311: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3i2.ccs.ornl.gov kernel: [1918860.235613] LustreError: 167-0: atlas2-MDT0000-lwp-OST03a1: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:54:57 atlas-oss3i2.ccs.ornl.gov kernel: [1918860.236231] Lustre: atlas2-MDT0000-lwp-OST0311: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:58 atlas-oss3i2.ccs.ornl.gov kernel: [1918860.316469] Lustre: atlas2-MDT0000-lwp-OST03a1: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:54:58 atlas-oss3i2.ccs.ornl.gov kernel: [1918860.346576] Lustre: Skipped 4 previous similar messages Jul 17 17:55:06 atlas-mds3.ccs.ornl.gov kernel: [ 674.286138] LustreError: 16151:0:(mdt_open.c:1504:mdt_reint_open()) @@@ OPEN & CREAT not in open replay/by_fid. req@ffff883fe531d400 x1472526902581756/t0(108775228021) o101->686195ed-cd47-0154-8d0f-2af8c9c96e26@10.38.145.23@o2ib4:0/0 lens 24736/3512 e 0 to 0 dl 1405634423 ref 1 fl Interpret:/4/0 rc 0/0 Jul 17 17:55:06 atlas-mds3.ccs.ornl.gov kernel: [ 674.316697] LustreError: 16151:0:(mdt_open.c:1504:mdt_reint_open()) Skipped 11 previous similar messages Jul 17 17:55:43 atlas-oss3i5.ccs.ornl.gov kernel: [705208.021298] LustreError: 167-0: atlas2-MDT0000-lwp-OST03a4: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:55:43 atlas-oss3i5.ccs.ornl.gov kernel: [705208.059527] Lustre: atlas2-MDT0000-lwp-OST03a4: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:55:43 atlas-oss3a8.ccs.ornl.gov kernel: [705208.947096] LustreError: 167-0: atlas2-MDT0000-lwp-OST0127: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:55:43 atlas-oss3i5.ccs.ornl.gov kernel: [705208.097618] Lustre: Skipped 5 previous similar messages Jul 17 17:55:43 atlas-oss3a8.ccs.ornl.gov kernel: [705208.981833] Lustre: atlas2-MDT0000-lwp-OST0127: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:55:43 atlas-oss3a8.ccs.ornl.gov kernel: [705209.011672] Lustre: Skipped 5 previous similar messages Jul 17 17:55:43 atlas-oss3d4.ccs.ornl.gov kernel: [705208.505663] LustreError: 167-0: atlas2-MDT0000-lwp-OST001b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:55:44 atlas-oss3d4.ccs.ornl.gov kernel: [705208.545542] Lustre: atlas2-MDT0000-lwp-OST025b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:55:44 atlas-oss3d4.ccs.ornl.gov kernel: [705208.583926] Lustre: Skipped 5 previous similar messages Jul 17 17:55:58 atlas-mds3.ccs.ornl.gov kernel: [ 726.304879] LustreError: 15401:0:(mdt_open.c:1504:mdt_reint_open()) @@@ OPEN & CREAT not in open replay/by_fid. req@ffff883fe07cc000 x1472463744890444/t0(108775228387) o101->02d34489-4d87-1e56-e805-ba4304f3351b@10.38.144.247@o2ib4:0/0 lens 24736/3512 e 0 to 0 dl 1405634475 ref 1 fl Interpret:/4/0 rc 0/0 Jul 17 17:55:58 atlas-mds3.ccs.ornl.gov kernel: [ 726.335516] LustreError: 15401:0:(mdt_open.c:1504:mdt_reint_open()) Skipped 23 previous similar messages Jul 17 17:56:58 atlas-mds3.ccs.ornl.gov kernel: [ 785.964695] LustreError: 15883:0:(mdt_open.c:1504:mdt_reint_open()) @@@ OPEN & CREAT not in open replay/by_fid. req@ffff883fdc01e000 x1473708410642828/t0(108666628734) o101->507c36d8-ed87-7d27-8cfb-03f4c9beded4@2284@gni100:0/0 lens 1024/3512 e 0 to 0 dl 1405634535 ref 1 fl Interpret:/4/0 rc 0/0 Jul 17 17:56:58 atlas-mds3.ccs.ornl.gov kernel: [ 785.994469] LustreError: 15883:0:(mdt_open.c:1504:mdt_reint_open()) Skipped 11 previous similar messages Jul 17 17:57:07 atlas-oss4b2.ccs.ornl.gov kernel: [1919006.422650] Lustre: 27461:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405633947/real 1405633947] req@ffff8808a5b38400 x1471904518026368/t0(0) o38->atlas2-MDT0000-lwp-OST0051@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:57:07 atlas-oss4b2.ccs.ornl.gov kernel: [1919006.513456] Lustre: 27461:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 17 17:57:07 atlas-oss3h5.ccs.ornl.gov kernel: [1919003.361370] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405633947/real 1405633947] req@ffff88075e214400 x1471904507807108/t0(0) o38->atlas2-MDT0000-lwp-OST003c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:57:07 atlas-oss3h5.ccs.ornl.gov kernel: [1919003.452652] Lustre: 24592:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 17 17:57:07 atlas-oss4c1.ccs.ornl.gov kernel: [1918989.260609] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405633947/real 1405633947] req@ffff88028651c800 x1471904516098024/t0(0) o38->atlas2-MDT0000-lwp-OST0058@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:57:07 atlas-oss3g7.ccs.ornl.gov kernel: [1917581.330270] Lustre: 13449:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405633947/real 1405633947] req@ffff8808709b6400 x1471904520422768/t0(0) o38->atlas2-MDT0000-lwp-OST0036@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:57:07 atlas-oss3g7.ccs.ornl.gov kernel: [1917581.421613] Lustre: 13449:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 17 17:57:07 atlas-oss4c1.ccs.ornl.gov kernel: [1918989.352171] Lustre: 22130:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 17 17:57:07 atlas-oss4f5.ccs.ornl.gov kernel: [1918981.728471] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405633947/real 1405633947] req@ffff880da69d1c00 x1471904509391448/t0(0) o38->atlas2-MDT0000-lwp-OST0074@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:57:07 atlas-oss4f5.ccs.ornl.gov kernel: [1918981.819148] Lustre: 22140:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 17 17:57:07 atlas-oss4g4.ccs.ornl.gov kernel: [1918998.323019] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405633947/real 1405633947] req@ffff8804332d3800 x1471904515531188/t0(0) o38->atlas2-MDT0000-lwp-OST007b@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:57:07 atlas-oss4g6.ccs.ornl.gov kernel: [1918993.553778] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405633947/real 1405633947] req@ffff8805ca7a5400 x1471904507899208/t0(0) o38->atlas2-MDT0000-lwp-OST007d@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:57:07 atlas-oss4g6.ccs.ornl.gov kernel: [1918993.645040] Lustre: 24689:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 17 17:57:07 atlas-oss4g7.ccs.ornl.gov kernel: [1919006.190987] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405633947/real 1405633947] req@ffff88097101d800 x1471904508874096/t0(0) o38->atlas2-MDT0000-lwp-OST007e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:57:07 atlas-oss4g7.ccs.ornl.gov kernel: [1919006.282069] Lustre: 26055:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 17 17:57:07 atlas-oss4g4.ccs.ornl.gov kernel: [1918998.414478] Lustre: 25810:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 17 17:57:07 atlas-oss4c7.ccs.ornl.gov kernel: [1918983.800600] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405633947/real 1405633947] req@ffff8804e3cdb000 x1471904514426132/t0(0) o38->atlas2-MDT0000-lwp-OST005e@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:57:07 atlas-oss4c7.ccs.ornl.gov kernel: [1918983.891864] Lustre: 25660:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Jul 17 17:57:07 atlas-oss4e5.ccs.ornl.gov kernel: [1918985.036656] Lustre: 24644:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405633947/real 1405633947] req@ffff8803eaf93400 x1471904515279724/t0(0) o38->atlas2-MDT0000-lwp-OST006c@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:57:07 atlas-oss4e5.ccs.ornl.gov kernel: [1918985.126875] Lustre: 24644:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 17 17:57:07 atlas-oss3e2.ccs.ornl.gov kernel: [1918983.402703] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405633947/real 1405633947] req@ffff8803c4607c00 x1471904513179712/t0(0) o38->atlas2-MDT0000-lwp-OST0381@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:57:07 atlas-oss3e2.ccs.ornl.gov kernel: [1918983.492096] Lustre: 27194:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Jul 17 17:57:07 atlas-oss3e6.ccs.ornl.gov kernel: [1918981.625779] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405633947/real 1405633947] req@ffff8805c87eac00 x1471904520535152/t0(0) o38->atlas2-MDT0000-lwp-OST0025@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:57:07 atlas-oss4b4.ccs.ornl.gov kernel: [1918982.163239] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405633947/real 1405633947] req@ffff8807092bcc00 x1471904528872532/t0(0) o38->atlas2-MDT0000-lwp-OST0053@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:57:07 atlas-oss4b4.ccs.ornl.gov kernel: [1918982.254272] Lustre: 26012:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 17 17:57:07 atlas-oss3b1.ccs.ornl.gov kernel: [1918990.749333] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1405633947/real 1405633947] req@ffff88032d10a800 x1471904516111264/t0(0) o38->atlas2-MDT0000-lwp-OST0008@10.36.226.74@o2ib:12/10 lens 400/544 e 0 to 1 dl 1405634227 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jul 17 17:57:07 atlas-oss3e6.ccs.ornl.gov kernel: [1918981.716510] Lustre: 22096:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 17 17:57:07 atlas-oss3b1.ccs.ornl.gov kernel: [1918990.840747] Lustre: 22311:0:(client.c:1868:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Jul 17 17:57:26 atlas-oss4g5.ccs.ornl.gov kernel: [1919003.506133] LustreError: 167-0: atlas2-MDT0000-lwp-OST007c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:26 atlas-oss4g5.ccs.ornl.gov kernel: [1919003.541589] LustreError: Skipped 4 previous similar messages Jul 17 17:57:26 atlas-oss4e4.ccs.ornl.gov kernel: [1919012.116028] LustreError: 167-0: atlas2-MDT0000-lwp-OST006b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:26 atlas-oss4e4.ccs.ornl.gov kernel: [1919012.150913] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4h7.ccs.ornl.gov kernel: [1919011.713798] LustreError: 167-0: atlas2-MDT0000-lwp-OST0356: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4h7.ccs.ornl.gov kernel: [1919011.747894] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss4d2.ccs.ornl.gov kernel: [1919008.528927] LustreError: 167-0: atlas2-MDT0000-lwp-OST0061: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4d2.ccs.ornl.gov kernel: [1919008.563708] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4g2.ccs.ornl.gov kernel: [1919014.202745] LustreError: 167-0: atlas2-MDT0000-lwp-OST0079: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4b3.ccs.ornl.gov kernel: [1918997.733005] LustreError: 167-0: atlas2-MDT0000-lwp-OST0052: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4b3.ccs.ornl.gov kernel: [1918997.766817] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4h6.ccs.ornl.gov kernel: [1919002.223763] LustreError: 167-0: atlas2-MDT0000-lwp-OST0085: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4h6.ccs.ornl.gov kernel: [1919002.264625] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4i7.ccs.ornl.gov kernel: [1919016.738379] LustreError: 167-0: atlas2-MDT0000-lwp-OST008e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4i7.ccs.ornl.gov kernel: [1919016.783134] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4i2.ccs.ornl.gov kernel: [1919026.341258] LustreError: 167-0: atlas2-MDT0000-lwp-OST0089: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4i2.ccs.ornl.gov kernel: [1919026.382606] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4f7.ccs.ornl.gov kernel: [1919002.512573] LustreError: 167-0: atlas2-MDT0000-lwp-OST0346: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4f7.ccs.ornl.gov kernel: [1919002.550357] LustreError: Skipped 2 previous similar messages Jul 17 17:57:27 atlas-oss4g2.ccs.ornl.gov kernel: [1919014.252774] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4e4.ccs.ornl.gov kernel: [1919012.171543] Lustre: atlas2-MDT0000-lwp-OST006b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4h2.ccs.ornl.gov kernel: [1919024.712529] LustreError: 167-0: atlas2-MDT0000-lwp-OST0081: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4h2.ccs.ornl.gov kernel: [1919024.754787] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4g5.ccs.ornl.gov kernel: [1919003.562250] Lustre: atlas2-MDT0000-lwp-OST007c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4b2.ccs.ornl.gov kernel: [1919026.308754] LustreError: 167-0: atlas2-MDT0000-lwp-OST0051: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4b2.ccs.ornl.gov kernel: [1919026.350906] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4h7.ccs.ornl.gov kernel: [1919011.768412] Lustre: atlas2-MDT0000-lwp-OST0086: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4h7.ccs.ornl.gov kernel: [1919011.806689] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss3e3.ccs.ornl.gov kernel: [1919000.160386] LustreError: 167-0: atlas2-MDT0000-lwp-OST0022: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3e3.ccs.ornl.gov kernel: [1919000.202417] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4d2.ccs.ornl.gov kernel: [1919008.584261] Lustre: atlas2-MDT0000-lwp-OST0061: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4h6.ccs.ornl.gov kernel: [1919002.284971] Lustre: atlas2-MDT0000-lwp-OST0085: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4b3.ccs.ornl.gov kernel: [1918997.787441] Lustre: atlas2-MDT0000-lwp-OST0052: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4g2.ccs.ornl.gov kernel: [1919014.273075] Lustre: atlas2-MDT0000-lwp-OST0079: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4i7.ccs.ornl.gov kernel: [1919016.803773] Lustre: atlas2-MDT0000-lwp-OST008e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4f7.ccs.ornl.gov kernel: [1919002.570953] Lustre: atlas2-MDT0000-lwp-OST0346: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4f7.ccs.ornl.gov kernel: [1919002.600818] Lustre: Skipped 2 previous similar messages Jul 17 17:57:27 atlas-oss4i2.ccs.ornl.gov kernel: [1919026.403078] Lustre: atlas2-MDT0000-lwp-OST0089: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4i3.ccs.ornl.gov kernel: [1918995.716856] LustreError: 167-0: atlas2-MDT0000-lwp-OST03ea: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4i3.ccs.ornl.gov kernel: [1918995.757029] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss4d7.ccs.ornl.gov kernel: [1919004.210865] LustreError: 167-0: atlas2-MDT0000-lwp-OST0066: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4d7.ccs.ornl.gov kernel: [1919004.251070] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4h2.ccs.ornl.gov kernel: [1919024.775206] Lustre: atlas2-MDT0000-lwp-OST0081: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3h5.ccs.ornl.gov kernel: [1919023.253127] LustreError: 167-0: atlas2-MDT0000-lwp-OST003c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4i4.ccs.ornl.gov kernel: [1919008.428629] LustreError: 167-0: atlas2-MDT0000-lwp-OST008b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3h5.ccs.ornl.gov kernel: [1919023.290354] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4i4.ccs.ornl.gov kernel: [1919008.464150] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3f6.ccs.ornl.gov kernel: [805708.106779] LustreError: 167-0: atlas2-MDT0000-lwp-OST002d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3f6.ccs.ornl.gov kernel: [805708.142095] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3c1.ccs.ornl.gov kernel: [1919005.655537] LustreError: 167-0: atlas2-MDT0000-lwp-OST0010: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3c1.ccs.ornl.gov kernel: [1919005.691287] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3i7.ccs.ornl.gov kernel: [1919000.670267] LustreError: 167-0: atlas2-MDT0000-lwp-OST03a6: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3i7.ccs.ornl.gov kernel: [1919000.705782] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss4b2.ccs.ornl.gov kernel: [1919026.371353] Lustre: atlas2-MDT0000-lwp-OST0051: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4f2.ccs.ornl.gov kernel: [1919007.099797] LustreError: 167-0: atlas2-MDT0000-lwp-OST0071: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4f2.ccs.ornl.gov kernel: [1919007.134844] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4d3.ccs.ornl.gov kernel: [1918999.996970] LustreError: 167-0: atlas2-MDT0000-lwp-OST03c2: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4d3.ccs.ornl.gov kernel: [1919000.034634] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss3e3.ccs.ornl.gov kernel: [1919000.223085] Lustre: atlas2-MDT0000-lwp-OST0022: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4c4.ccs.ornl.gov kernel: [1919015.419188] LustreError: 167-0: atlas2-MDT0000-lwp-OST005b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4h4.ccs.ornl.gov kernel: [1919022.977148] LustreError: 167-0: atlas2-MDT0000-lwp-OST0083: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4h4.ccs.ornl.gov kernel: [1919023.014507] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4c5.ccs.ornl.gov kernel: [1918999.277812] LustreError: 167-0: atlas2-MDT0000-lwp-OST005c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4c5.ccs.ornl.gov kernel: [1918999.317904] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4c3.ccs.ornl.gov kernel: [1919000.319324] LustreError: 167-0: atlas2-MDT0000-lwp-OST005a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4c3.ccs.ornl.gov kernel: [1919000.354148] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3g8.ccs.ornl.gov kernel: [1917601.604744] LustreError: 167-0: atlas2-MDT0000-lwp-OST0037: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3g8.ccs.ornl.gov kernel: [1917601.638969] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4i3.ccs.ornl.gov kernel: [1918995.777480] Lustre: atlas2-MDT0000-lwp-OST03ea: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4i3.ccs.ornl.gov kernel: [1918995.807481] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss3h8.ccs.ornl.gov kernel: [1919000.417327] LustreError: 167-0: atlas2-MDT0000-lwp-OST003f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3h8.ccs.ornl.gov kernel: [1919000.453376] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3f4.ccs.ornl.gov kernel: [1919005.726432] LustreError: 167-0: atlas2-MDT0000-lwp-OST002b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3f4.ccs.ornl.gov kernel: [1919005.762302] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4c4.ccs.ornl.gov kernel: [1919015.459889] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4a4.ccs.ornl.gov kernel: [1919005.110697] LustreError: 167-0: atlas2-MDT0000-lwp-OST004b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4a4.ccs.ornl.gov kernel: [1919005.149623] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3h5.ccs.ornl.gov kernel: [1919023.310845] Lustre: atlas2-MDT0000-lwp-OST003c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3f6.ccs.ornl.gov kernel: [805708.162724] Lustre: atlas2-MDT0000-lwp-OST002d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4d7.ccs.ornl.gov kernel: [1919004.271724] Lustre: atlas2-MDT0000-lwp-OST0066: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3d6.ccs.ornl.gov kernel: [1919017.967359] LustreError: 167-0: atlas2-MDT0000-lwp-OST001d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3d6.ccs.ornl.gov kernel: [1919018.018912] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3c1.ccs.ornl.gov kernel: [1919005.711737] Lustre: atlas2-MDT0000-lwp-OST0010: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4a5.ccs.ornl.gov kernel: [1919022.566965] LustreError: 167-0: atlas2-MDT0000-lwp-OST004c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4a5.ccs.ornl.gov kernel: [1919022.608649] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4i4.ccs.ornl.gov kernel: [1919008.484838] Lustre: atlas2-MDT0000-lwp-OST008b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3i7.ccs.ornl.gov kernel: [1919000.726304] Lustre: atlas2-MDT0000-lwp-OST03a6: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3i7.ccs.ornl.gov kernel: [1919000.756164] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss4f2.ccs.ornl.gov kernel: [1919007.155308] Lustre: atlas2-MDT0000-lwp-OST0071: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4c8.ccs.ornl.gov kernel: [1918995.433576] LustreError: 167-0: atlas2-MDT0000-lwp-OST005f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3e1.ccs.ornl.gov kernel: [1919006.329063] LustreError: 167-0: atlas2-MDT0000-lwp-OST0020: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4c8.ccs.ornl.gov kernel: [1918995.469049] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3e1.ccs.ornl.gov kernel: [1919006.368432] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4d3.ccs.ornl.gov kernel: [1919000.055122] Lustre: atlas2-MDT0000-lwp-OST03c2: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4d3.ccs.ornl.gov kernel: [1919000.085086] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss4e7.ccs.ornl.gov kernel: [1919000.978517] LustreError: 167-0: atlas2-MDT0000-lwp-OST03ce: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4e7.ccs.ornl.gov kernel: [1919001.018188] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss3f5.ccs.ornl.gov kernel: [805695.465587] LustreError: 167-0: atlas2-MDT0000-lwp-OST002c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3f5.ccs.ornl.gov kernel: [805695.506353] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss4a2.ccs.ornl.gov kernel: [1919018.232536] LustreError: 167-0: atlas2-MDT0000-lwp-OST0049: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4a2.ccs.ornl.gov kernel: [1919018.274432] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4c4.ccs.ornl.gov kernel: [1919015.480137] Lustre: atlas2-MDT0000-lwp-OST005b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3f8.ccs.ornl.gov kernel: [805705.980474] LustreError: 167-0: atlas2-MDT0000-lwp-OST002f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3f8.ccs.ornl.gov kernel: [805706.022367] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4h4.ccs.ornl.gov kernel: [1919023.035124] Lustre: atlas2-MDT0000-lwp-OST0083: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4c3.ccs.ornl.gov kernel: [1919000.374653] Lustre: atlas2-MDT0000-lwp-OST005a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4c2.ccs.ornl.gov kernel: [1919007.530638] LustreError: 167-0: atlas2-MDT0000-lwp-OST0059: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4c2.ccs.ornl.gov kernel: [1919007.567461] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4c5.ccs.ornl.gov kernel: [1918999.338609] Lustre: atlas2-MDT0000-lwp-OST005c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3g5.ccs.ornl.gov kernel: [1917605.190489] LustreError: 167-0: atlas2-MDT0000-lwp-OST0034: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3g5.ccs.ornl.gov kernel: [1917605.232577] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3f4.ccs.ornl.gov kernel: [1919005.782783] Lustre: atlas2-MDT0000-lwp-OST002b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3g8.ccs.ornl.gov kernel: [1917601.659439] Lustre: atlas2-MDT0000-lwp-OST0037: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4a4.ccs.ornl.gov kernel: [1919005.170119] Lustre: atlas2-MDT0000-lwp-OST004b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3h8.ccs.ornl.gov kernel: [1919000.474016] Lustre: atlas2-MDT0000-lwp-OST003f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3d6.ccs.ornl.gov kernel: [1919018.039343] Lustre: atlas2-MDT0000-lwp-OST001d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4e1.ccs.ornl.gov kernel: [1919001.472218] LustreError: 167-0: atlas2-MDT0000-lwp-OST0068: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4a5.ccs.ornl.gov kernel: [1919022.629298] Lustre: atlas2-MDT0000-lwp-OST004c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4e1.ccs.ornl.gov kernel: [1919001.512021] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3h3.ccs.ornl.gov kernel: [1918999.221222] LustreError: 167-0: atlas2-MDT0000-lwp-OST003a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3h3.ccs.ornl.gov kernel: [1918999.255492] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4d4.ccs.ornl.gov kernel: [1919024.006784] LustreError: 167-0: atlas2-MDT0000-lwp-OST0063: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4d4.ccs.ornl.gov kernel: [1919024.041423] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3h7.ccs.ornl.gov kernel: [1919010.362277] LustreError: 167-0: atlas2-MDT0000-lwp-OST003e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4g8.ccs.ornl.gov kernel: [1919002.721320] LustreError: 167-0: atlas2-MDT0000-lwp-OST007f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4g8.ccs.ornl.gov kernel: [1919002.759050] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3h7.ccs.ornl.gov kernel: [1919010.395800] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3a1.ccs.ornl.gov kernel: [1918998.448302] LustreError: 167-0: atlas2-MDT0000-lwp-OST0000: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3a1.ccs.ornl.gov kernel: [1918998.485143] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4e7.ccs.ornl.gov kernel: [1919001.038696] Lustre: atlas2-MDT0000-lwp-OST03ce: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4e7.ccs.ornl.gov kernel: [1919001.068523] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss3f5.ccs.ornl.gov kernel: [805695.526782] Lustre: atlas2-MDT0000-lwp-OST002c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3f5.ccs.ornl.gov kernel: [805695.556607] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss4a2.ccs.ornl.gov kernel: [1919018.294891] Lustre: atlas2-MDT0000-lwp-OST0049: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3e1.ccs.ornl.gov kernel: [1919006.388976] Lustre: atlas2-MDT0000-lwp-OST0020: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4c8.ccs.ornl.gov kernel: [1918995.489732] Lustre: atlas2-MDT0000-lwp-OST005f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3f7.ccs.ornl.gov kernel: [805334.712239] LustreError: 167-0: atlas2-MDT0000-lwp-OST002e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3f7.ccs.ornl.gov kernel: [805334.750545] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3f8.ccs.ornl.gov kernel: [805706.043101] Lustre: atlas2-MDT0000-lwp-OST002f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4e2.ccs.ornl.gov kernel: [1919017.467296] LustreError: 167-0: atlas2-MDT0000-lwp-OST0069: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4e2.ccs.ornl.gov kernel: [1919017.507940] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss4c1.ccs.ornl.gov kernel: [1919009.154362] LustreError: 167-0: atlas2-MDT0000-lwp-OST0058: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4c1.ccs.ornl.gov kernel: [1919009.189894] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4h8.ccs.ornl.gov kernel: [1918994.573690] LustreError: 167-0: atlas2-MDT0000-lwp-OST0087: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4h8.ccs.ornl.gov kernel: [1918994.615604] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4b6.ccs.ornl.gov kernel: [1919001.253882] LustreError: 167-0: atlas2-MDT0000-lwp-OST0055: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4b6.ccs.ornl.gov kernel: [1919001.291031] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4c2.ccs.ornl.gov kernel: [1919007.588116] Lustre: atlas2-MDT0000-lwp-OST0059: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3g7.ccs.ornl.gov kernel: [1917601.215958] LustreError: 167-0: atlas2-MDT0000-lwp-OST0036: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3g7.ccs.ornl.gov kernel: [1917601.249313] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3g5.ccs.ornl.gov kernel: [1917605.253237] Lustre: atlas2-MDT0000-lwp-OST0034: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3b3.ccs.ornl.gov kernel: [1918996.615021] LustreError: 167-0: atlas2-MDT0000-lwp-OST000a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3b3.ccs.ornl.gov kernel: [1918996.648782] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4b1.ccs.ornl.gov kernel: [1919008.158021] LustreError: 167-0: atlas2-MDT0000-lwp-OST0050: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4b1.ccs.ornl.gov kernel: [1919008.195035] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4e1.ccs.ornl.gov kernel: [1919001.532917] Lustre: atlas2-MDT0000-lwp-OST0068: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3h3.ccs.ornl.gov kernel: [1918999.275786] Lustre: atlas2-MDT0000-lwp-OST003a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3a1.ccs.ornl.gov kernel: [1918998.505627] Lustre: atlas2-MDT0000-lwp-OST0000: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3f7.ccs.ornl.gov kernel: [805334.770890] Lustre: atlas2-MDT0000-lwp-OST002e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4d4.ccs.ornl.gov kernel: [1919024.061940] Lustre: atlas2-MDT0000-lwp-OST0063: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4g8.ccs.ornl.gov kernel: [1919002.779631] Lustre: atlas2-MDT0000-lwp-OST007f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3h7.ccs.ornl.gov kernel: [1919010.416636] Lustre: atlas2-MDT0000-lwp-OST003e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4e2.ccs.ornl.gov kernel: [1919017.528267] Lustre: atlas2-MDT0000-lwp-OST0069: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4e2.ccs.ornl.gov kernel: [1919017.558239] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss4g3.ccs.ornl.gov kernel: [1919014.845552] LustreError: 167-0: atlas2-MDT0000-lwp-OST007a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4f5.ccs.ornl.gov kernel: [1919001.621177] LustreError: 167-0: atlas2-MDT0000-lwp-OST0074: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4f5.ccs.ornl.gov kernel: [1919001.656815] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3a6.ccs.ornl.gov kernel: [1919006.327373] LustreError: 167-0: atlas2-MDT0000-lwp-OST0005: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3a5.ccs.ornl.gov kernel: [1918997.215637] LustreError: 167-0: atlas2-MDT0000-lwp-OST0004: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3a6.ccs.ornl.gov kernel: [1919006.363187] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3a5.ccs.ornl.gov kernel: [1918997.255126] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss4c1.ccs.ornl.gov kernel: [1919009.210015] Lustre: atlas2-MDT0000-lwp-OST0058: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4h8.ccs.ornl.gov kernel: [1918994.636252] Lustre: atlas2-MDT0000-lwp-OST0087: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3b3.ccs.ornl.gov kernel: [1918996.669312] Lustre: atlas2-MDT0000-lwp-OST000a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4b1.ccs.ornl.gov kernel: [1919008.215332] Lustre: atlas2-MDT0000-lwp-OST0050: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4b6.ccs.ornl.gov kernel: [1919001.311681] Lustre: atlas2-MDT0000-lwp-OST0055: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3g7.ccs.ornl.gov kernel: [1917601.270143] Lustre: atlas2-MDT0000-lwp-OST0036: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4g3.ccs.ornl.gov kernel: [1919014.886393] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3b5.ccs.ornl.gov kernel: [1919016.757892] LustreError: 167-0: atlas2-MDT0000-lwp-OST000c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4f6.ccs.ornl.gov kernel: [1918998.499710] LustreError: 167-0: atlas2-MDT0000-lwp-OST0075: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3b5.ccs.ornl.gov kernel: [1919016.796225] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4f6.ccs.ornl.gov kernel: [1918998.536294] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3i4.ccs.ornl.gov kernel: [1919009.603935] LustreError: 167-0: atlas2-MDT0000-lwp-OST0043: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3i4.ccs.ornl.gov kernel: [1919009.646161] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4g6.ccs.ornl.gov kernel: [1919013.439650] LustreError: 167-0: atlas2-MDT0000-lwp-OST007d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4g6.ccs.ornl.gov kernel: [1919013.473244] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4g7.ccs.ornl.gov kernel: [1919026.072804] LustreError: 167-0: atlas2-MDT0000-lwp-OST007e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4g4.ccs.ornl.gov kernel: [1919018.208936] LustreError: 167-0: atlas2-MDT0000-lwp-OST007b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4g7.ccs.ornl.gov kernel: [1919026.110285] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4g3.ccs.ornl.gov kernel: [1919014.906899] Lustre: atlas2-MDT0000-lwp-OST007a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4f5.ccs.ornl.gov kernel: [1919001.677290] Lustre: atlas2-MDT0000-lwp-OST0074: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3a6.ccs.ornl.gov kernel: [1919006.383717] Lustre: atlas2-MDT0000-lwp-OST0005: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3a5.ccs.ornl.gov kernel: [1918997.275819] Lustre: atlas2-MDT0000-lwp-OST0364: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3a5.ccs.ornl.gov kernel: [1918997.305602] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss4f3.ccs.ornl.gov kernel: [1918998.456793] LustreError: 167-0: atlas2-MDT0000-lwp-OST0072: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4b7.ccs.ornl.gov kernel: [1918997.801660] LustreError: 167-0: atlas2-MDT0000-lwp-OST0056: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4f3.ccs.ornl.gov kernel: [1918998.494452] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4b7.ccs.ornl.gov kernel: [1918997.840590] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3h2.ccs.ornl.gov kernel: [1919006.789705] LustreError: 167-0: atlas2-MDT0000-lwp-OST0039: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3h2.ccs.ornl.gov kernel: [1919006.831568] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3h4.ccs.ornl.gov kernel: [1919007.530463] LustreError: 167-0: atlas2-MDT0000-lwp-OST003b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3h4.ccs.ornl.gov kernel: [1919007.571759] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4b8.ccs.ornl.gov kernel: [1919005.611356] LustreError: 167-0: atlas2-MDT0000-lwp-OST0057: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4b8.ccs.ornl.gov kernel: [1919005.647796] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4g4.ccs.ornl.gov kernel: [1919018.251078] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3e7.ccs.ornl.gov kernel: [1919006.155143] LustreError: 167-0: atlas2-MDT0000-lwp-OST0026: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3e7.ccs.ornl.gov kernel: [1919006.193035] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3g1.ccs.ornl.gov kernel: [1917593.712605] LustreError: 167-0: atlas2-MDT0000-lwp-OST0030: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3g1.ccs.ornl.gov kernel: [1917593.752186] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3a2.ccs.ornl.gov kernel: [1919003.298984] LustreError: 167-0: atlas2-MDT0000-lwp-OST0361: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3a2.ccs.ornl.gov kernel: [1919003.336438] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss3g2.ccs.ornl.gov kernel: [1917600.402482] LustreError: 167-0: atlas2-MDT0000-lwp-OST0031: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3g2.ccs.ornl.gov kernel: [1917600.444324] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3h1.ccs.ornl.gov kernel: [1918992.466452] LustreError: 167-0: atlas2-MDT0000-lwp-OST0038: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3h1.ccs.ornl.gov kernel: [1918992.501504] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4f6.ccs.ornl.gov kernel: [1918998.556697] Lustre: atlas2-MDT0000-lwp-OST0075: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3b5.ccs.ornl.gov kernel: [1919016.816704] Lustre: atlas2-MDT0000-lwp-OST000c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3i4.ccs.ornl.gov kernel: [1919009.658329] Lustre: atlas2-MDT0000-lwp-OST0043: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4e8.ccs.ornl.gov kernel: [1919016.755522] LustreError: 167-0: atlas2-MDT0000-lwp-OST03cf: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4e8.ccs.ornl.gov kernel: [1919016.796347] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss4i8.ccs.ornl.gov kernel: [1919017.502368] LustreError: 167-0: atlas2-MDT0000-lwp-OST008f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4i8.ccs.ornl.gov kernel: [1919017.535884] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4e3.ccs.ornl.gov kernel: [1919009.608350] LustreError: 167-0: atlas2-MDT0000-lwp-OST006a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4e3.ccs.ornl.gov kernel: [1919009.647133] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4g7.ccs.ornl.gov kernel: [1919026.130889] Lustre: atlas2-MDT0000-lwp-OST007e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4g4.ccs.ornl.gov kernel: [1919018.263275] Lustre: atlas2-MDT0000-lwp-OST007b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4f1.ccs.ornl.gov kernel: [1919007.970547] LustreError: 167-0: atlas2-MDT0000-lwp-OST0070: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4b7.ccs.ornl.gov kernel: [1918997.861140] Lustre: atlas2-MDT0000-lwp-OST0056: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4f3.ccs.ornl.gov kernel: [1918998.514957] Lustre: atlas2-MDT0000-lwp-OST0072: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4d5.ccs.ornl.gov kernel: [1919016.313437] LustreError: 167-0: atlas2-MDT0000-lwp-OST0064: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4g6.ccs.ornl.gov kernel: [1919013.493888] Lustre: atlas2-MDT0000-lwp-OST007d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4i5.ccs.ornl.gov kernel: [1919011.762220] LustreError: 167-0: atlas2-MDT0000-lwp-OST008c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4i5.ccs.ornl.gov kernel: [1919011.799144] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3i8.ccs.ornl.gov kernel: [1919017.717314] LustreError: 167-0: atlas2-MDT0000-lwp-OST0047: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3i8.ccs.ornl.gov kernel: [1919017.755157] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3h2.ccs.ornl.gov kernel: [1919006.852239] Lustre: atlas2-MDT0000-lwp-OST0039: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3h4.ccs.ornl.gov kernel: [1919007.592229] Lustre: atlas2-MDT0000-lwp-OST003b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3d8.ccs.ornl.gov kernel: [1919010.378493] LustreError: 167-0: atlas2-MDT0000-lwp-OST037f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3f2.ccs.ornl.gov kernel: [1918998.938791] LustreError: 167-0: atlas2-MDT0000-lwp-OST0029: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3f2.ccs.ornl.gov kernel: [1918998.974991] LustreError: Skipped 2 previous similar messages Jul 17 17:57:27 atlas-oss3e7.ccs.ornl.gov kernel: [1919006.213523] Lustre: atlas2-MDT0000-lwp-OST0026: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4b8.ccs.ornl.gov kernel: [1919005.668727] Lustre: atlas2-MDT0000-lwp-OST0057: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3g1.ccs.ornl.gov kernel: [1917593.772693] Lustre: atlas2-MDT0000-lwp-OST0030: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4h5.ccs.ornl.gov kernel: [1919006.323174] LustreError: 167-0: atlas2-MDT0000-lwp-OST0084: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4h5.ccs.ornl.gov kernel: [1919006.364485] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4f1.ccs.ornl.gov kernel: [1919008.012770] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4d5.ccs.ornl.gov kernel: [1919016.360469] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3a2.ccs.ornl.gov kernel: [1919003.356929] Lustre: atlas2-MDT0000-lwp-OST0361: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3a4.ccs.ornl.gov kernel: [1918993.785923] LustreError: 167-0: atlas2-MDT0000-lwp-OST0003: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3a2.ccs.ornl.gov kernel: [1919003.386729] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss3a4.ccs.ornl.gov kernel: [1918993.825027] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3g2.ccs.ornl.gov kernel: [1917600.464637] Lustre: atlas2-MDT0000-lwp-OST0031: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4c7.ccs.ornl.gov kernel: [1919003.690261] LustreError: 167-0: atlas2-MDT0000-lwp-OST005e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4b5.ccs.ornl.gov kernel: [1919021.414492] LustreError: 167-0: atlas2-MDT0000-lwp-OST0054: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4b5.ccs.ornl.gov kernel: [1919021.452927] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3d8.ccs.ornl.gov kernel: [1919010.414270] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss3h1.ccs.ornl.gov kernel: [1918992.522019] Lustre: atlas2-MDT0000-lwp-OST0038: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3d1.ccs.ornl.gov kernel: [1918996.487094] LustreError: 167-0: atlas2-MDT0000-lwp-OST0018: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3d1.ccs.ornl.gov kernel: [1918996.524443] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4i6.ccs.ornl.gov kernel: [1919017.231229] LustreError: 167-0: atlas2-MDT0000-lwp-OST008d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4i6.ccs.ornl.gov kernel: [1919017.270929] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4h1.ccs.ornl.gov kernel: [1918997.700166] LustreError: 167-0: atlas2-MDT0000-lwp-OST0080: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4e5.ccs.ornl.gov kernel: [1919004.930368] LustreError: 167-0: atlas2-MDT0000-lwp-OST006c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4e5.ccs.ornl.gov kernel: [1919004.964510] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3e8.ccs.ornl.gov kernel: [1919003.950899] LustreError: 167-0: atlas2-MDT0000-lwp-OST0027: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3e8.ccs.ornl.gov kernel: [1919003.988590] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3b6.ccs.ornl.gov kernel: [1919002.061200] LustreError: 167-0: atlas2-MDT0000-lwp-OST000d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3b6.ccs.ornl.gov kernel: [1919002.099831] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4f4.ccs.ornl.gov kernel: [1919004.293561] LustreError: 167-0: atlas2-MDT0000-lwp-OST0073: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4f4.ccs.ornl.gov kernel: [1919004.331466] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4c7.ccs.ornl.gov kernel: [1919003.729537] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3b2.ccs.ornl.gov kernel: [1919003.839850] LustreError: 167-0: atlas2-MDT0000-lwp-OST0009: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4e3.ccs.ornl.gov kernel: [1919009.667820] Lustre: atlas2-MDT0000-lwp-OST006a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3b2.ccs.ornl.gov kernel: [1919003.877783] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4a6.ccs.ornl.gov kernel: [1919005.919166] LustreError: 167-0: atlas2-MDT0000-lwp-OST004d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4a6.ccs.ornl.gov kernel: [1919005.953698] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3a3.ccs.ornl.gov kernel: [1919001.069902] LustreError: 167-0: atlas2-MDT0000-lwp-OST0002: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4e8.ccs.ornl.gov kernel: [1919016.817122] Lustre: atlas2-MDT0000-lwp-OST03cf: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3a3.ccs.ornl.gov kernel: [1919001.111814] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4f1.ccs.ornl.gov kernel: [1919008.033197] Lustre: atlas2-MDT0000-lwp-OST0070: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4e8.ccs.ornl.gov kernel: [1919016.855194] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss4i8.ccs.ornl.gov kernel: [1919017.556573] Lustre: atlas2-MDT0000-lwp-OST008f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4i5.ccs.ornl.gov kernel: [1919011.819652] Lustre: atlas2-MDT0000-lwp-OST008c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4h1.ccs.ornl.gov kernel: [1918997.740103] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3e5.ccs.ornl.gov kernel: [1919005.092847] LustreError: 167-0: atlas2-MDT0000-lwp-OST0024: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3e5.ccs.ornl.gov kernel: [1919005.135128] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4d6.ccs.ornl.gov kernel: [1919003.946306] LustreError: 167-0: atlas2-MDT0000-lwp-OST0065: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4d6.ccs.ornl.gov kernel: [1919003.981738] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3d8.ccs.ornl.gov kernel: [1919010.434622] Lustre: atlas2-MDT0000-lwp-OST037f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3d8.ccs.ornl.gov kernel: [1919010.464422] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss4d5.ccs.ornl.gov kernel: [1919016.381105] Lustre: atlas2-MDT0000-lwp-OST0064: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3i8.ccs.ornl.gov kernel: [1919017.775788] Lustre: atlas2-MDT0000-lwp-OST0047: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3c5.ccs.ornl.gov kernel: [1919010.483382] LustreError: 167-0: atlas2-MDT0000-lwp-OST0014: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3c5.ccs.ornl.gov kernel: [1919010.519676] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3g4.ccs.ornl.gov kernel: [1917614.384553] LustreError: 167-0: atlas2-MDT0000-lwp-OST0033: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3g4.ccs.ornl.gov kernel: [1917614.426453] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss3f2.ccs.ornl.gov kernel: [1918998.995725] Lustre: atlas2-MDT0000-lwp-OST0029: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3f2.ccs.ornl.gov kernel: [1918999.025380] Lustre: Skipped 2 previous similar messages Jul 17 17:57:27 atlas-oss3c2.ccs.ornl.gov kernel: [1919005.913349] LustreError: 167-0: atlas2-MDT0000-lwp-OST0011: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3c2.ccs.ornl.gov kernel: [1919005.955403] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4g1.ccs.ornl.gov kernel: [1919000.936294] LustreError: 167-0: atlas2-MDT0000-lwp-OST0078: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4g1.ccs.ornl.gov kernel: [1919000.978299] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4h5.ccs.ornl.gov kernel: [1919006.385154] Lustre: atlas2-MDT0000-lwp-OST0084: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3f3.ccs.ornl.gov kernel: [1919017.413223] LustreError: 167-0: atlas2-MDT0000-lwp-OST002a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3f3.ccs.ornl.gov kernel: [1919017.452983] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3a4.ccs.ornl.gov kernel: [1918993.845403] Lustre: atlas2-MDT0000-lwp-OST0003: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3c6.ccs.ornl.gov kernel: [1918996.119642] LustreError: 167-0: atlas2-MDT0000-lwp-OST0015: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3c6.ccs.ornl.gov kernel: [1918996.153085] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3d1.ccs.ornl.gov kernel: [1918996.544905] Lustre: atlas2-MDT0000-lwp-OST0018: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3c7.ccs.ornl.gov kernel: [1919004.750470] LustreError: 167-0: atlas2-MDT0000-lwp-OST0376: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3c7.ccs.ornl.gov kernel: [1919004.791806] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss4c7.ccs.ornl.gov kernel: [1919003.750085] Lustre: atlas2-MDT0000-lwp-OST005e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4i6.ccs.ornl.gov kernel: [1919017.291463] Lustre: atlas2-MDT0000-lwp-OST008d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4b5.ccs.ornl.gov kernel: [1919021.473713] Lustre: atlas2-MDT0000-lwp-OST0054: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4h1.ccs.ornl.gov kernel: [1918997.760476] Lustre: atlas2-MDT0000-lwp-OST0080: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3e8.ccs.ornl.gov kernel: [1919004.009232] Lustre: atlas2-MDT0000-lwp-OST0027: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3b6.ccs.ornl.gov kernel: [1919002.120327] Lustre: atlas2-MDT0000-lwp-OST000d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3e4.ccs.ornl.gov kernel: [1919018.157050] LustreError: 167-0: atlas2-MDT0000-lwp-OST0023: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3e4.ccs.ornl.gov kernel: [1919018.198860] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4c6.ccs.ornl.gov kernel: [1919004.806634] LustreError: 167-0: atlas2-MDT0000-lwp-OST005d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4c6.ccs.ornl.gov kernel: [1919004.843171] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3b2.ccs.ornl.gov kernel: [1919003.898392] Lustre: atlas2-MDT0000-lwp-OST0009: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4e5.ccs.ornl.gov kernel: [1919004.984897] Lustre: atlas2-MDT0000-lwp-OST006c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3d3.ccs.ornl.gov kernel: [1919010.727904] LustreError: 167-0: atlas2-MDT0000-lwp-OST001a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3d3.ccs.ornl.gov kernel: [1919010.769984] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4f4.ccs.ornl.gov kernel: [1919004.352132] Lustre: atlas2-MDT0000-lwp-OST0073: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4a3.ccs.ornl.gov kernel: [1919019.881354] LustreError: 167-0: atlas2-MDT0000-lwp-OST004a: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3a3.ccs.ornl.gov kernel: [1919001.132271] Lustre: atlas2-MDT0000-lwp-OST0002: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4a3.ccs.ornl.gov kernel: [1919019.923699] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3c8.ccs.ornl.gov kernel: [1919008.946429] LustreError: 167-0: atlas2-MDT0000-lwp-OST0017: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3c8.ccs.ornl.gov kernel: [1919008.987038] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4a1.ccs.ornl.gov kernel: [1919001.842341] LustreError: 167-0: atlas2-MDT0000-lwp-OST0048: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4a7.ccs.ornl.gov kernel: [1919015.576342] LustreError: 167-0: atlas2-MDT0000-lwp-OST004e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4e6.ccs.ornl.gov kernel: [1919019.685168] LustreError: 167-0: atlas2-MDT0000-lwp-OST006d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4e6.ccs.ornl.gov kernel: [1919019.720739] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3e2.ccs.ornl.gov kernel: [1919003.288328] LustreError: 167-0: atlas2-MDT0000-lwp-OST0381: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3e2.ccs.ornl.gov kernel: [1919003.329438] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss4a6.ccs.ornl.gov kernel: [1919005.974373] Lustre: atlas2-MDT0000-lwp-OST004d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4d6.ccs.ornl.gov kernel: [1919004.002296] Lustre: atlas2-MDT0000-lwp-OST0065: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3c5.ccs.ornl.gov kernel: [1919010.540174] Lustre: atlas2-MDT0000-lwp-OST0014: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3g4.ccs.ornl.gov kernel: [1917614.447072] Lustre: atlas2-MDT0000-lwp-OST0033: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3g4.ccs.ornl.gov kernel: [1917614.476824] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss3h6.ccs.ornl.gov kernel: [1919004.288438] LustreError: 167-0: atlas2-MDT0000-lwp-OST003d: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3h6.ccs.ornl.gov kernel: [1919004.328475] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3c2.ccs.ornl.gov kernel: [1919005.975692] Lustre: atlas2-MDT0000-lwp-OST0011: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4g1.ccs.ornl.gov kernel: [1919000.998790] Lustre: atlas2-MDT0000-lwp-OST0078: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3e5.ccs.ornl.gov kernel: [1919005.155632] Lustre: atlas2-MDT0000-lwp-OST0024: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3d2.ccs.ornl.gov kernel: [1919005.627676] LustreError: 167-0: atlas2-MDT0000-lwp-OST0379: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3d2.ccs.ornl.gov kernel: [1919005.662529] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss4a1.ccs.ornl.gov kernel: [1919001.880512] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4a7.ccs.ornl.gov kernel: [1919015.618482] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3i6.ccs.ornl.gov kernel: [1919002.773168] LustreError: 167-0: atlas2-MDT0000-lwp-OST0045: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3f3.ccs.ornl.gov kernel: [1919017.473853] Lustre: atlas2-MDT0000-lwp-OST002a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3c4.ccs.ornl.gov kernel: [1918999.206607] LustreError: 167-0: atlas2-MDT0000-lwp-OST0013: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3c4.ccs.ornl.gov kernel: [1918999.248468] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3g3.ccs.ornl.gov kernel: [1917601.323327] LustreError: 167-0: atlas2-MDT0000-lwp-OST0392: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3g3.ccs.ornl.gov kernel: [1917601.361316] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss3c7.ccs.ornl.gov kernel: [1919004.812434] Lustre: atlas2-MDT0000-lwp-OST0376: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3c7.ccs.ornl.gov kernel: [1919004.842365] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss3c6.ccs.ornl.gov kernel: [1918996.173760] Lustre: atlas2-MDT0000-lwp-OST0015: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4d1.ccs.ornl.gov kernel: [1918999.955015] LustreError: 167-0: atlas2-MDT0000-lwp-OST0060: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4d1.ccs.ornl.gov kernel: [1918999.993982] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3a7.ccs.ornl.gov kernel: [1919002.061512] LustreError: 167-0: atlas2-MDT0000-lwp-OST0006: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3a7.ccs.ornl.gov kernel: [1919002.096131] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3i3.ccs.ornl.gov kernel: [1918999.404319] LustreError: 167-0: atlas2-MDT0000-lwp-OST0042: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3i3.ccs.ornl.gov kernel: [1918999.440768] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3i6.ccs.ornl.gov kernel: [1919002.812544] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3g6.ccs.ornl.gov kernel: [1917612.339847] LustreError: 167-0: atlas2-MDT0000-lwp-OST0035: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3g6.ccs.ornl.gov kernel: [1917612.377861] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3d3.ccs.ornl.gov kernel: [1919010.790462] Lustre: atlas2-MDT0000-lwp-OST001a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4a3.ccs.ornl.gov kernel: [1919019.943997] Lustre: atlas2-MDT0000-lwp-OST004a: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3c8.ccs.ornl.gov kernel: [1919009.007412] Lustre: atlas2-MDT0000-lwp-OST0017: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3e4.ccs.ornl.gov kernel: [1919018.219415] Lustre: atlas2-MDT0000-lwp-OST0023: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4a1.ccs.ornl.gov kernel: [1919001.901181] Lustre: atlas2-MDT0000-lwp-OST0048: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4c6.ccs.ornl.gov kernel: [1919004.863793] Lustre: atlas2-MDT0000-lwp-OST005d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3e6.ccs.ornl.gov kernel: [1919001.519409] LustreError: 167-0: atlas2-MDT0000-lwp-OST0025: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3e6.ccs.ornl.gov kernel: [1919001.553845] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4a7.ccs.ornl.gov kernel: [1919015.639242] Lustre: atlas2-MDT0000-lwp-OST004e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4e6.ccs.ornl.gov kernel: [1919019.741385] Lustre: atlas2-MDT0000-lwp-OST006d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4b4.ccs.ornl.gov kernel: [1919002.056859] LustreError: 167-0: atlas2-MDT0000-lwp-OST0053: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4b4.ccs.ornl.gov kernel: [1919002.091903] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3e2.ccs.ornl.gov kernel: [1919003.350208] Lustre: atlas2-MDT0000-lwp-OST0381: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3e2.ccs.ornl.gov kernel: [1919003.379652] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss4h3.ccs.ornl.gov kernel: [1919016.655250] LustreError: 167-0: atlas2-MDT0000-lwp-OST0082: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4h3.ccs.ornl.gov kernel: [1919016.697275] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3c3.ccs.ornl.gov kernel: [1918996.632334] LustreError: 167-0: atlas2-MDT0000-lwp-OST0012: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3c3.ccs.ornl.gov kernel: [1918996.673619] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss3d7.ccs.ornl.gov kernel: [1919004.823332] LustreError: 167-0: atlas2-MDT0000-lwp-OST001e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3d7.ccs.ornl.gov kernel: [1919004.864116] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4i1.ccs.ornl.gov kernel: [1919004.069809] LustreError: 167-0: atlas2-MDT0000-lwp-OST0088: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4i1.ccs.ornl.gov kernel: [1919004.111833] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4a8.ccs.ornl.gov kernel: [1919010.010996] LustreError: 167-0: atlas2-MDT0000-lwp-OST004f: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4a8.ccs.ornl.gov kernel: [1919010.051525] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3b7.ccs.ornl.gov kernel: [1919008.858167] LustreError: 167-0: atlas2-MDT0000-lwp-OST000e: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3b7.ccs.ornl.gov kernel: [1919008.891976] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss3f1.ccs.ornl.gov kernel: [1919007.801742] LustreError: 167-0: atlas2-MDT0000-lwp-OST0028: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3f1.ccs.ornl.gov kernel: [1919007.843507] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4f8.ccs.ornl.gov kernel: [1919003.211073] LustreError: 167-0: atlas2-MDT0000-lwp-OST0077: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3d2.ccs.ornl.gov kernel: [1919005.683155] Lustre: atlas2-MDT0000-lwp-OST0379: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3d2.ccs.ornl.gov kernel: [1919005.713180] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss3h6.ccs.ornl.gov kernel: [1919004.349139] Lustre: atlas2-MDT0000-lwp-OST003d: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3b1.ccs.ornl.gov kernel: [1919010.639153] LustreError: 167-0: atlas2-MDT0000-lwp-OST0008: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3b1.ccs.ornl.gov kernel: [1919010.678427] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3g3.ccs.ornl.gov kernel: [1917601.381799] Lustre: atlas2-MDT0000-lwp-OST0032: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3i1.ccs.ornl.gov kernel: [1919004.131639] LustreError: 167-0: atlas2-MDT0000-lwp-OST03a0: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3i1.ccs.ornl.gov kernel: [1919004.167879] LustreError: Skipped 3 previous similar messages Jul 17 17:57:27 atlas-oss3i6.ccs.ornl.gov kernel: [1919002.832885] Lustre: atlas2-MDT0000-lwp-OST0045: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4d1.ccs.ornl.gov kernel: [1919000.014475] Lustre: atlas2-MDT0000-lwp-OST0060: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4f8.ccs.ornl.gov kernel: [1919003.253023] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3i3.ccs.ornl.gov kernel: [1918999.461527] Lustre: atlas2-MDT0000-lwp-OST0042: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3c4.ccs.ornl.gov kernel: [1918999.269105] Lustre: atlas2-MDT0000-lwp-OST0013: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3a7.ccs.ornl.gov kernel: [1919002.116603] Lustre: atlas2-MDT0000-lwp-OST0006: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3g6.ccs.ornl.gov kernel: [1917612.398243] Lustre: atlas2-MDT0000-lwp-OST0035: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4d8.ccs.ornl.gov kernel: [1919017.067319] LustreError: 167-0: atlas2-MDT0000-lwp-OST0067: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss4d8.ccs.ornl.gov kernel: [1919017.109174] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3g3.ccs.ornl.gov kernel: [1917601.411710] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss4h3.ccs.ornl.gov kernel: [1919016.717817] Lustre: atlas2-MDT0000-lwp-OST0082: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3c3.ccs.ornl.gov kernel: [1918996.694334] Lustre: atlas2-MDT0000-lwp-OST0012: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3c3.ccs.ornl.gov kernel: [1918996.724279] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss4a8.ccs.ornl.gov kernel: [1919010.072030] Lustre: atlas2-MDT0000-lwp-OST004f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4i1.ccs.ornl.gov kernel: [1919004.132529] Lustre: atlas2-MDT0000-lwp-OST0088: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3e6.ccs.ornl.gov kernel: [1919001.574683] Lustre: atlas2-MDT0000-lwp-OST0025: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3b8.ccs.ornl.gov kernel: [1919018.193894] LustreError: 167-0: atlas2-MDT0000-lwp-OST02df: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3b8.ccs.ornl.gov kernel: [1919018.227590] LustreError: Skipped 2 previous similar messages Jul 17 17:57:27 atlas-oss4b4.ccs.ornl.gov kernel: [1919002.112729] Lustre: atlas2-MDT0000-lwp-OST0053: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3f1.ccs.ornl.gov kernel: [1919007.864195] Lustre: atlas2-MDT0000-lwp-OST0028: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3d7.ccs.ornl.gov kernel: [1919004.884793] Lustre: atlas2-MDT0000-lwp-OST001e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss4f8.ccs.ornl.gov kernel: [1919003.273516] Lustre: atlas2-MDT0000-lwp-OST0077: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3b7.ccs.ornl.gov kernel: [1919008.912455] Lustre: atlas2-MDT0000-lwp-OST000e: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3b7.ccs.ornl.gov kernel: [1919008.950993] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss3b1.ccs.ornl.gov kernel: [1919010.698906] Lustre: atlas2-MDT0000-lwp-OST0008: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3i1.ccs.ornl.gov kernel: [1919004.188483] Lustre: atlas2-MDT0000-lwp-OST03a0: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3i1.ccs.ornl.gov kernel: [1919004.226761] Lustre: Skipped 1 previous similar message Jul 17 17:57:27 atlas-oss3d5.ccs.ornl.gov kernel: [1919007.533596] LustreError: 167-0: atlas2-MDT0000-lwp-OST001c: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3d5.ccs.ornl.gov kernel: [1919007.568681] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss4d8.ccs.ornl.gov kernel: [1919017.129847] Lustre: atlas2-MDT0000-lwp-OST0067: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3b8.ccs.ornl.gov kernel: [1919018.248014] Lustre: atlas2-MDT0000-lwp-OST000f: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3b8.ccs.ornl.gov kernel: [1919018.278149] Lustre: Skipped 2 previous similar messages Jul 17 17:57:27 atlas-oss3d5.ccs.ornl.gov kernel: [1919007.589176] Lustre: atlas2-MDT0000-lwp-OST001c: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3b4.ccs.ornl.gov kernel: [1919015.713451] LustreError: 167-0: atlas2-MDT0000-lwp-OST000b: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3b4.ccs.ornl.gov kernel: [1919015.748801] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3b4.ccs.ornl.gov kernel: [1919015.769511] Lustre: atlas2-MDT0000-lwp-OST000b: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:57:27 atlas-oss3i2.ccs.ornl.gov kernel: [1919010.249688] LustreError: 167-0: atlas2-MDT0000-lwp-OST0041: This client was evicted by atlas2-MDT0000; in progress operations using this service will fail. Jul 17 17:57:27 atlas-oss3i2.ccs.ornl.gov kernel: [1919010.283624] LustreError: Skipped 4 previous similar messages Jul 17 17:57:27 atlas-oss3i2.ccs.ornl.gov kernel: [1919010.304223] Lustre: atlas2-MDT0000-lwp-OST0041: Connection restored to atlas2-MDT0000 (at 10.36.226.74@o2ib) Jul 17 17:58:28 atlas-mds3.ccs.ornl.gov kernel: [ 876.536895] Lustre: atlas2-MDT0000: Client 54f5a678-687f-943e-dd74-d7ac707f4a2d (at 91@gni2) reconnecting, waiting for 20156 clients in recovery for 24:38 Jul 17 17:58:28 atlas-mds3.ccs.ornl.gov kernel: [ 876.552685] Lustre: atlas2-MDT0000: Client 54f5a678-687f-943e-dd74-d7ac707f4a2d (at 91@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 17:58:28 atlas-mds3.ccs.ornl.gov kernel: [ 876.577502] Lustre: atlas2-MDT0000: Client c2999fec-1da2-9f46-3cb4-7ad1d4f1dacd (at 89@gni2) reconnecting, waiting for 20156 clients in recovery for 24:38 Jul 17 17:58:28 atlas-mds3.ccs.ornl.gov kernel: [ 876.593281] Lustre: atlas2-MDT0000: Client c2999fec-1da2-9f46-3cb4-7ad1d4f1dacd (at 89@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 17:58:29 atlas-mds3.ccs.ornl.gov kernel: [ 877.298818] Lustre: atlas2-MDT0000: Client 953c9541-2381-eb2f-3b54-fc227e440056 (at 90@gni2) reconnecting, waiting for 20156 clients in recovery for 24:38 Jul 17 17:58:29 atlas-mds3.ccs.ornl.gov kernel: [ 877.312165] Lustre: atlas2-MDT0000: Client 8801c3e0-18d6-cd99-1113-65baae637edf (at 51@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 17:58:29 atlas-mds3.ccs.ornl.gov kernel: [ 877.312168] Lustre: Skipped 5 previous similar messages Jul 17 17:58:29 atlas-mds3.ccs.ornl.gov kernel: [ 877.335372] Lustre: Skipped 6 previous similar messages Jul 17 17:58:30 atlas-mds3.ccs.ornl.gov kernel: [ 878.401346] Lustre: atlas2-MDT0000: Client 27f41f9f-3331-87d2-d5cd-588e13f9c6b6 (at 10@gni2) reconnecting, waiting for 20156 clients in recovery for 24:37 Jul 17 17:58:30 atlas-mds3.ccs.ornl.gov kernel: [ 878.416250] Lustre: atlas2-MDT0000: Client a7561335-42cd-e1cc-d1a4-6096eb4c9aab (at 78@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 17:58:30 atlas-mds3.ccs.ornl.gov kernel: [ 878.416252] Lustre: Skipped 20 previous similar messages Jul 17 17:58:30 atlas-mds3.ccs.ornl.gov kernel: [ 878.438028] Lustre: Skipped 20 previous similar messages Jul 17 17:58:32 atlas-mds3.ccs.ornl.gov kernel: [ 880.470956] Lustre: atlas2-MDT0000: Client 7c50d969-127e-e39a-27ed-e6ccd617b844 (at 50@gni2) reconnecting, waiting for 20156 clients in recovery for 24:35 Jul 17 17:58:32 atlas-mds3.ccs.ornl.gov kernel: [ 880.486738] Lustre: Skipped 27 previous similar messages Jul 17 17:58:32 atlas-mds3.ccs.ornl.gov kernel: [ 880.492877] Lustre: atlas2-MDT0000: Client 7c50d969-127e-e39a-27ed-e6ccd617b844 (at 50@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 17:58:32 atlas-mds3.ccs.ornl.gov kernel: [ 880.507687] Lustre: Skipped 28 previous similar messages Jul 17 17:58:39 atlas-mds3.ccs.ornl.gov kernel: [ 887.679482] Lustre: atlas2-MDT0000: Client 25eca314-2213-dba4-4e7c-ae1acb406793 (at 15@gni2) reconnecting, waiting for 20156 clients in recovery for 24:27 Jul 17 17:58:39 atlas-mds3.ccs.ornl.gov kernel: [ 887.695233] Lustre: Skipped 20 previous similar messages Jul 17 17:58:39 atlas-mds3.ccs.ornl.gov kernel: [ 887.701359] Lustre: atlas2-MDT0000: Client 25eca314-2213-dba4-4e7c-ae1acb406793 (at 15@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 17:58:39 atlas-mds3.ccs.ornl.gov kernel: [ 887.716136] Lustre: Skipped 20 previous similar messages Jul 17 18:00:46 atlas-mds3.ccs.ornl.gov kernel: [ 1013.988416] Lustre: atlas2-MDT0000: Client 84eb7626-264a-9db5-eff3-ee82928c6490 (at 44@gni2) reconnecting, waiting for 20156 clients in recovery for 22:21 Jul 17 18:00:46 atlas-mds3.ccs.ornl.gov kernel: [ 1014.004171] Lustre: Skipped 1 previous similar message Jul 17 18:00:46 atlas-mds3.ccs.ornl.gov kernel: [ 1014.010108] Lustre: atlas2-MDT0000: Client 84eb7626-264a-9db5-eff3-ee82928c6490 (at 44@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 18:00:46 atlas-mds3.ccs.ornl.gov kernel: [ 1014.024877] Lustre: Skipped 1 previous similar message Jul 17 18:02:38 atlas-mds3.ccs.ornl.gov kernel: [ 1126.006188] Lustre: atlas2-MDT0000: Client 3de4154a-ce79-b7fc-0e3e-2c2d35ec9598 (at 10.36.202.130@o2ib) reconnecting, waiting for 20156 clients in recovery for 20:29 Jul 17 18:02:38 atlas-mds3.ccs.ornl.gov kernel: [ 1126.023141] Lustre: Skipped 20 previous similar messages Jul 17 18:02:38 atlas-mds3.ccs.ornl.gov kernel: [ 1126.029290] Lustre: atlas2-MDT0000: Client 3de4154a-ce79-b7fc-0e3e-2c2d35ec9598 (at 10.36.202.130@o2ib) refused reconnection, still busy with 1 active RPCs Jul 17 18:02:38 atlas-mds3.ccs.ornl.gov kernel: [ 1126.045173] Lustre: Skipped 20 previous similar messages Jul 17 18:03:16 atlas-mds3.ccs.ornl.gov kernel: [ 1164.133940] Lustre: atlas2-MDT0000: Client 84eb7626-264a-9db5-eff3-ee82928c6490 (at 44@gni2) reconnecting, waiting for 20156 clients in recovery for 19:51 Jul 17 18:03:16 atlas-mds3.ccs.ornl.gov kernel: [ 1164.149700] Lustre: Skipped 15 previous similar messages Jul 17 18:03:16 atlas-mds3.ccs.ornl.gov kernel: [ 1164.155821] Lustre: atlas2-MDT0000: Client 84eb7626-264a-9db5-eff3-ee82928c6490 (at 44@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 18:03:16 atlas-mds3.ccs.ornl.gov kernel: [ 1164.170603] Lustre: Skipped 10 previous similar messages Jul 17 18:03:18 atlas-mds3.ccs.ornl.gov kernel: [ 1166.556076] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff881e364586c0/0xf35a0587b959b549 lrc: 3/0,0 mode: PR/PR res: [0x2000d1f90:0x2fa5:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 9650@gni100 remote: 0xc421be983f83a635 expref: 128 pid: 15432 timeout: 0 lvb_type: 0 used 0 Jul 17 18:05:46 atlas-mds3.ccs.ornl.gov kernel: [ 1314.277675] Lustre: atlas2-MDT0000: Client 84eb7626-264a-9db5-eff3-ee82928c6490 (at 44@gni2) reconnecting, waiting for 20156 clients in recovery for 17:21 Jul 17 18:05:46 atlas-mds3.ccs.ornl.gov kernel: [ 1314.293369] Lustre: Skipped 82 previous similar messages Jul 17 18:05:46 atlas-mds3.ccs.ornl.gov kernel: [ 1314.299442] Lustre: atlas2-MDT0000: Client 84eb7626-264a-9db5-eff3-ee82928c6490 (at 44@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 18:05:46 atlas-mds3.ccs.ornl.gov kernel: [ 1314.314179] Lustre: Skipped 80 previous similar messages Jul 17 18:08:16 atlas-mds3.ccs.ornl.gov kernel: [ 1464.435036] Lustre: atlas2-MDT0000: Client 84eb7626-264a-9db5-eff3-ee82928c6490 (at 44@gni2) reconnecting, waiting for 20156 clients in recovery for 14:51 Jul 17 18:08:16 atlas-mds3.ccs.ornl.gov kernel: [ 1464.450720] Lustre: Skipped 94 previous similar messages Jul 17 18:08:16 atlas-mds3.ccs.ornl.gov kernel: [ 1464.456784] Lustre: atlas2-MDT0000: Client 84eb7626-264a-9db5-eff3-ee82928c6490 (at 44@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 18:08:16 atlas-mds3.ccs.ornl.gov kernel: [ 1464.471488] Lustre: Skipped 93 previous similar messages Jul 17 18:09:34 atlas-mds3.ccs.ornl.gov kernel: [ 1542.388438] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 9650@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff881e364586c0/0xf35a0587b959b549 lrc: 3/0,0 mode: PR/PR res: [0x2000d1f90:0x2fa5:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x20 nid: 9650@gni100 remote: 0xc421be983f83a635 expref: 128 pid: 15432 timeout: 4296206555 lvb_type: 0 used 0 Jul 17 18:09:35 atlas-mds3.ccs.ornl.gov kernel: [ 1543.577807] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff883fcfbcc000/0xf35a0587b94e5271 lrc: 3/0,0 mode: PR/PR res: [0x2000d42bc:0x1f84:0x0].0 bits 0x13 rrc: 3 type: IBT flags: 0x20 nid: 7124@gni100 remote: 0xca8480f794af6a9 expref: 127 pid: 16174 timeout: 0 lvb_type: 0 used 0 Jul 17 18:13:07 atlas-mds1.ccs.ornl.gov kernel: [799958.300312] LustreError: 22496:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 198215277639, new: 198215277638 replay: 0. see LU-617. Jul 17 18:13:07 atlas-mds1.ccs.ornl.gov kernel: [799958.318233] LustreError: 22496:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Skipped 9 previous similar messages Jul 17 18:13:07 atlas-mds1.ccs.ornl.gov kernel: [799958.377611] LustreError: 22472:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 198215278645, new: 198215278644 replay: 0. see LU-617. Jul 17 18:13:07 atlas-mds1.ccs.ornl.gov kernel: [799958.888906] LustreError: 22472:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 198215281848, new: 198215281847 replay: 0. see LU-617. Jul 17 18:13:16 atlas-mds3.ccs.ornl.gov kernel: [ 1764.708817] Lustre: atlas2-MDT0000: Client 84eb7626-264a-9db5-eff3-ee82928c6490 (at 44@gni2) reconnecting, waiting for 20156 clients in recovery for 9:55 Jul 17 18:13:16 atlas-mds3.ccs.ornl.gov kernel: [ 1764.724404] Lustre: Skipped 183 previous similar messages Jul 17 18:13:16 atlas-mds3.ccs.ornl.gov kernel: [ 1764.730557] Lustre: atlas2-MDT0000: Client 84eb7626-264a-9db5-eff3-ee82928c6490 (at 44@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 18:13:16 atlas-mds3.ccs.ornl.gov kernel: [ 1764.745268] Lustre: Skipped 183 previous similar messages Jul 17 18:15:49 atlas-mds3.ccs.ornl.gov kernel: [ 1918.193476] Lustre: atlas2-MDT0000: Denying connection for new client 31afef4f-2324-fa17-9f8f-aa171f72e2b3 (at 9650@gni100), waiting for all 20156 known clients (18665 recovered, 1490 in progress, and 1 evicted) to recover in 7:22 Jul 17 18:15:51 atlas-mds3.ccs.ornl.gov kernel: [ 1919.742426] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 7124@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff883fcfbcc000/0xf35a0587b94e5271 lrc: 3/0,0 mode: PR/PR res: [0x2000d42bc:0x1f84:0x0].0 bits 0x13 rrc: 3 type: IBT flags: 0x20 nid: 7124@gni100 remote: 0xca8480f794af6a9 expref: 127 pid: 16174 timeout: 4296583226 lvb_type: 0 used 0 Jul 17 18:15:52 atlas-mds3.ccs.ornl.gov kernel: [ 1920.718968] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff881d43d2b240/0xf35a0587ba949a45 lrc: 3/0,0 mode: PR/PR res: [0x2000d1fb3:0x345c:0x0].0 bits 0x13 rrc: 3 type: IBT flags: 0x20 nid: 11532@gni100 remote: 0x1a8f46db5213208a expref: 128 pid: 15584 timeout: 0 lvb_type: 0 used 0 Jul 17 18:21:59 atlas-mds3.ccs.ornl.gov kernel: [ 2288.173289] Lustre: atlas2-MDT0000: Client 88af36eb-fc99-9aa3-c80e-6b143a6ac03a (at 10.36.225.5@o2ib) reconnecting, waiting for 20156 clients in recovery for 7:29 Jul 17 18:21:59 atlas-mds3.ccs.ornl.gov kernel: [ 2288.189850] Lustre: Skipped 360 previous similar messages Jul 17 18:21:59 atlas-mds3.ccs.ornl.gov kernel: [ 2288.196006] Lustre: atlas2-MDT0000: Client 88af36eb-fc99-9aa3-c80e-6b143a6ac03a (at 10.36.225.5@o2ib) refused reconnection, still busy with 1 active RPCs Jul 17 18:21:59 atlas-mds3.ccs.ornl.gov kernel: [ 2288.211592] Lustre: Skipped 360 previous similar messages Jul 17 18:22:08 atlas-mds3.ccs.ornl.gov kernel: [ 2297.099771] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 11532@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff881d43d2b240/0xf35a0587ba949a45 lrc: 3/0,0 mode: PR/PR res: [0x2000d1fb3:0x345c:0x0].0 bits 0x13 rrc: 3 type: IBT flags: 0x20 nid: 11532@gni100 remote: 0x1a8f46db5213208a expref: 128 pid: 15584 timeout: 4296960013 lvb_type: 0 used 0 Jul 17 18:22:08 atlas-mds3.ccs.ornl.gov kernel: [ 2297.308875] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff883fc4cdf240/0xf35a0587b954759d lrc: 3/0,0 mode: PR/PR res: [0x2000d248a:0x4d4f:0x0].0 bits 0x13 rrc: 3 type: IBT flags: 0x20 nid: 2760@gni100 remote: 0x2f556210427fd73d expref: 129 pid: 16173 timeout: 0 lvb_type: 0 used 0 Jul 17 18:23:58 atlas-mds3.ccs.ornl.gov kernel: [ 2408.127275] Lustre: atlas2-MDT0000: Denying connection for new client b30cd215-376e-3ae1-3b74-f835fde13fd4 (at 7124@gni100), waiting for all 20156 known clients (18792 recovered, 1361 in progress, and 3 evicted) to recover in 11:46 Jul 17 18:25:00 atlas-mds3.ccs.ornl.gov kernel: [ 2469.719827] Lustre: atlas2-MDT0000: Denying connection for new client 31afef4f-2324-fa17-9f8f-aa171f72e2b3 (at 9650@gni100), waiting for all 20156 known clients (18792 recovered, 1361 in progress, and 3 evicted) to recover in 10:44 Jul 17 18:28:24 atlas-mds3.ccs.ornl.gov kernel: [ 2673.461545] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 2760@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff883fc4cdf240/0xf35a0587b954759d lrc: 3/0,0 mode: PR/PR res: [0x2000d248a:0x4d4f:0x0].0 bits 0x13 rrc: 3 type: IBT flags: 0x20 nid: 2760@gni100 remote: 0x2f556210427fd73d expref: 129 pid: 16173 timeout: 4297336247 lvb_type: 0 used 0 Jul 17 18:30:30 atlas-mds3.ccs.ornl.gov kernel: [ 2799.560401] Lustre: atlas2-MDT0000: Denying connection for new client 4b75194a-e7f4-0e32-26c4-09848435aa4d (at 11532@gni100), waiting for all 20156 known clients (18830 recovered, 1322 in progress, and 4 evicted) to recover in 11:31 Jul 17 18:30:47 atlas-mds3.ccs.ornl.gov kernel: [ 2817.166669] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff881d44464d80/0xf35a0587ba95995c lrc: 3/0,0 mode: PR/PR res: [0x2000d19f5:0xda0:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x20 nid: 7131@gni100 remote: 0xda39f0601204c57f expref: 130 pid: 15658 timeout: 0 lvb_type: 0 used 0 Jul 17 18:32:21 atlas-mds3.ccs.ornl.gov kernel: [ 2911.599678] Lustre: atlas2-MDT0000: Client f04d019e-f170-2481-7488-6181a0e28e3e (at 10.36.202.176@o2ib) reconnecting, waiting for 20156 clients in recovery for 12:02 Jul 17 18:32:21 atlas-mds3.ccs.ornl.gov kernel: [ 2911.602699] Lustre: atlas2-MDT0000: Client 1841c050-d33f-3b1f-84ac-2fe38e66471e (at 10.36.202.175@o2ib) refused reconnection, still busy with 1 active RPCs Jul 17 18:32:21 atlas-mds3.ccs.ornl.gov kernel: [ 2911.602701] Lustre: Skipped 365 previous similar messages Jul 17 18:32:21 atlas-mds3.ccs.ornl.gov kernel: [ 2911.638517] Lustre: Skipped 368 previous similar messages Jul 17 18:33:09 atlas-mds3.ccs.ornl.gov kernel: [ 2959.654623] Lustre: atlas2-MDT0000: Denying connection for new client b30cd215-376e-3ae1-3b74-f835fde13fd4 (at 7124@gni100), waiting for all 20156 known clients (18835 recovered, 1317 in progress, and 4 evicted) to recover in 11:14 Jul 17 18:34:11 atlas-mds3.ccs.ornl.gov kernel: [ 3021.247082] Lustre: atlas2-MDT0000: Denying connection for new client 31afef4f-2324-fa17-9f8f-aa171f72e2b3 (at 9650@gni100), waiting for all 20156 known clients (18835 recovered, 1317 in progress, and 4 evicted) to recover in 10:13 Jul 17 18:36:14 atlas-mds3.ccs.ornl.gov kernel: [ 3144.138626] Lustre: atlas2-MDT0000: Denying connection for new client e7e497f0-c155-31e1-a97b-4cbe4568679c (at 2760@gni100), waiting for all 20156 known clients (18835 recovered, 1317 in progress, and 4 evicted) to recover in 8:10 Jul 17 18:37:03 atlas-mds3.ccs.ornl.gov kernel: [ 3192.958486] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 7131@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff881d44464d80/0xf35a0587ba95995c lrc: 3/0,0 mode: PR/PR res: [0x2000d19f5:0xda0:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x20 nid: 7131@gni100 remote: 0xda39f0601204c57f expref: 130 pid: 15658 timeout: 4297855605 lvb_type: 0 used 0 Jul 17 18:37:03 atlas-mds3.ccs.ornl.gov kernel: [ 3193.224433] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff881e30d98900/0xf35a0587ba95d80f lrc: 3/0,0 mode: PR/PR res: [0x2000d39fe:0x4716:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x20 nid: 1080@gni100 remote: 0xcd5c7d0e47f66b8b expref: 126 pid: 15701 timeout: 0 lvb_type: 0 used 0 Jul 17 18:39:41 atlas-mds3.ccs.ornl.gov kernel: [ 3351.086754] Lustre: atlas2-MDT0000: Denying connection for new client 4b75194a-e7f4-0e32-26c4-09848435aa4d (at 11532@gni100), waiting for all 20156 known clients (18857 recovered, 1294 in progress, and 5 evicted) to recover in 10:59 Jul 17 18:42:20 atlas-mds3.ccs.ornl.gov kernel: [ 3511.182590] Lustre: atlas2-MDT0000: Denying connection for new client b30cd215-376e-3ae1-3b74-f835fde13fd4 (at 7124@gni100), waiting for all 20156 known clients (18857 recovered, 1294 in progress, and 5 evicted) to recover in 8:19 Jul 17 18:42:22 atlas-mds3.ccs.ornl.gov kernel: [ 3512.223408] Lustre: atlas2-MDT0000: Client 619b2c60-515d-a8a9-0a1f-9d0e5f389969 (at 10.36.202.170@o2ib) reconnecting, waiting for 20156 clients in recovery for 8:18 Jul 17 18:42:22 atlas-mds3.ccs.ornl.gov kernel: [ 3512.227602] Lustre: atlas2-MDT0000: Client 4d283764-21de-ce01-b405-1aa66230b48a (at 10.36.202.171@o2ib) refused reconnection, still busy with 1 active RPCs Jul 17 18:42:22 atlas-mds3.ccs.ornl.gov kernel: [ 3512.227604] Lustre: Skipped 371 previous similar messages Jul 17 18:42:22 atlas-mds3.ccs.ornl.gov kernel: [ 3512.262069] Lustre: Skipped 371 previous similar messages Jul 17 18:43:19 atlas-mds3.ccs.ornl.gov kernel: [ 3569.318011] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 1080@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff881e30d98900/0xf35a0587ba95d80f lrc: 3/0,0 mode: PR/PR res: [0x2000d39fe:0x4716:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x20 nid: 1080@gni100 remote: 0xcd5c7d0e47f66b8b expref: 126 pid: 15701 timeout: 4298231303 lvb_type: 0 used 0 Jul 17 18:43:19 atlas-mds3.ccs.ornl.gov kernel: [ 3569.619763] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff881d0870db40/0xf35a0587ba96b9c8 lrc: 3/0,0 mode: PR/PR res: [0x2000d28bd:0x19d3:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x20 nid: 11386@gni100 remote: 0xb86f34d0d1dbdfb1 expref: 131 pid: 15452 timeout: 0 lvb_type: 0 used 0 Jul 17 18:44:09 atlas-mds1.ccs.ornl.gov kernel: [801822.546467] LustreError: 22489:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 198224443838, new: 198224443837 replay: 0. see LU-617. Jul 17 18:44:09 atlas-mds1.ccs.ornl.gov kernel: [801822.564389] LustreError: 22489:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Skipped 10 previous similar messages Jul 17 18:45:25 atlas-mds3.ccs.ornl.gov kernel: [ 3695.666805] Lustre: atlas2-MDT0000: Denying connection for new client e7e497f0-c155-31e1-a97b-4cbe4568679c (at 2760@gni100), waiting for all 20156 known clients (18890 recovered, 1260 in progress, and 6 evicted) to recover in 11:30 Jul 17 18:45:25 atlas-mds3.ccs.ornl.gov kernel: [ 3695.690081] Lustre: Skipped 2 previous similar messages Jul 17 18:48:52 atlas-mds3.ccs.ornl.gov kernel: [ 3902.613828] Lustre: atlas2-MDT0000: Denying connection for new client 4b75194a-e7f4-0e32-26c4-09848435aa4d (at 11532@gni100), waiting for all 20156 known clients (18890 recovered, 1260 in progress, and 6 evicted) to recover in 8:04 Jul 17 18:49:35 atlas-mds3.ccs.ornl.gov kernel: [ 3945.677112] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 11386@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff881d0870db40/0xf35a0587ba96b9c8 lrc: 3/0,0 mode: PR/PR res: [0x2000d28bd:0x19d3:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x20 nid: 11386@gni100 remote: 0xb86f34d0d1dbdfb1 expref: 131 pid: 15452 timeout: 4298607339 lvb_type: 0 used 0 Jul 17 18:49:35 atlas-mds3.ccs.ornl.gov kernel: [ 3945.769562] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff883fc5b45d80/0xf35a0587ba96e160 lrc: 3/0,0 mode: PR/PR res: [0x2000d38af:0xd6c:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x20 nid: 5559@gni100 remote: 0xb7cc4ce86be0843 expref: 130 pid: 15946 timeout: 0 lvb_type: 0 used 0 Jul 17 18:52:22 atlas-mds3.ccs.ornl.gov kernel: [ 4112.815408] Lustre: atlas2-MDT0000: Client 1b5cd858-3437-a120-43eb-465c5e782a10 (at 10.36.202.177@o2ib) reconnecting, waiting for 20156 clients in recovery for 10:50 Jul 17 18:52:22 atlas-mds3.ccs.ornl.gov kernel: [ 4112.832270] Lustre: Skipped 374 previous similar messages Jul 17 18:52:22 atlas-mds3.ccs.ornl.gov kernel: [ 4112.838429] Lustre: atlas2-MDT0000: Client 1b5cd858-3437-a120-43eb-465c5e782a10 (at 10.36.202.177@o2ib) refused reconnection, still busy with 1 active RPCs Jul 17 18:52:22 atlas-mds3.ccs.ornl.gov kernel: [ 4112.854233] Lustre: Skipped 376 previous similar messages Jul 17 18:54:36 atlas-mds3.ccs.ornl.gov kernel: [ 4247.195965] Lustre: atlas2-MDT0000: Denying connection for new client e7e497f0-c155-31e1-a97b-4cbe4568679c (at 2760@gni100), waiting for all 20156 known clients (18897 recovered, 1252 in progress, and 7 evicted) to recover in 8:35 Jul 17 18:54:36 atlas-mds3.ccs.ornl.gov kernel: [ 4247.219144] Lustre: Skipped 4 previous similar messages Jul 17 18:55:51 atlas-mds3.ccs.ornl.gov kernel: [ 4322.038018] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 5559@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff883fc5b45d80/0xf35a0587ba96e160 lrc: 3/0,0 mode: PR/PR res: [0x2000d38af:0xd6c:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x20 nid: 5559@gni100 remote: 0xb7cc4ce86be0843 expref: 130 pid: 15946 timeout: 4298983130 lvb_type: 0 used 0 Jul 17 18:55:51 atlas-mds3.ccs.ornl.gov kernel: [ 4322.152070] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff881d1deb4240/0xf35a0587ba9652b9 lrc: 3/0,0 mode: PR/PR res: [0x2000d47f3:0x94a:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x20 nid: 11440@gni100 remote: 0x56434e3f35ec2b36 expref: 131 pid: 15651 timeout: 0 lvb_type: 0 used 0 Jul 17 19:02:07 atlas-mds3.ccs.ornl.gov kernel: [ 4698.397653] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 11440@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff881d1deb4240/0xf35a0587ba9652b9 lrc: 3/0,0 mode: PR/PR res: [0x2000d47f3:0x94a:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x20 nid: 11440@gni100 remote: 0x56434e3f35ec2b36 expref: 131 pid: 15651 timeout: 4299359152 lvb_type: 0 used 0 Jul 17 19:02:07 atlas-mds3.ccs.ornl.gov kernel: [ 4698.471879] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff881d0854f000/0xf35a0587ba96c487 lrc: 3/0,0 mode: PR/PR res: [0x2000d3b5a:0x338a:0x0].0 bits 0x13 rrc: 9 type: IBT flags: 0x20 nid: 8173@gni100 remote: 0x28f5e6aebf088c7f expref: 127 pid: 15688 timeout: 0 lvb_type: 0 used 0 Jul 17 19:02:22 atlas-mds3.ccs.ornl.gov kernel: [ 4713.398195] Lustre: atlas2-MDT0000: Client a6f0eb44-8c3f-0211-09d3-6939a864b79b (at 10.36.202.173@o2ib) reconnecting, waiting for 20156 clients in recovery for 13:22 Jul 17 19:02:22 atlas-mds3.ccs.ornl.gov kernel: [ 4713.415053] Lustre: Skipped 370 previous similar messages Jul 17 19:02:22 atlas-mds3.ccs.ornl.gov kernel: [ 4713.421209] Lustre: atlas2-MDT0000: Client a6f0eb44-8c3f-0211-09d3-6939a864b79b (at 10.36.202.173@o2ib) refused reconnection, still busy with 1 active RPCs Jul 17 19:02:22 atlas-mds3.ccs.ornl.gov kernel: [ 4713.436981] Lustre: Skipped 369 previous similar messages Jul 17 19:03:47 atlas-mds3.ccs.ornl.gov kernel: [ 4798.725004] Lustre: atlas2-MDT0000: Denying connection for new client e7e497f0-c155-31e1-a97b-4cbe4568679c (at 2760@gni100), waiting for all 20156 known clients (18918 recovered, 1229 in progress, and 9 evicted) to recover in 11:56 Jul 17 19:03:47 atlas-mds3.ccs.ornl.gov kernel: [ 4798.748290] Lustre: Skipped 6 previous similar messages Jul 17 19:08:23 atlas-mds3.ccs.ornl.gov kernel: [ 5074.759521] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 8173@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff881d0854f000/0xf35a0587ba96c487 lrc: 3/0,0 mode: PR/PR res: [0x2000d3b5a:0x338a:0x0].0 bits 0x13 rrc: 9 type: IBT flags: 0x20 nid: 8173@gni100 remote: 0x28f5e6aebf088c7f expref: 127 pid: 15688 timeout: 4299735112 lvb_type: 0 used 0 Jul 17 19:08:23 atlas-mds3.ccs.ornl.gov kernel: [ 5074.864514] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff881e3927e900/0xf35a0587ba95dbea lrc: 3/0,0 mode: PR/PR res: [0x2000d270e:0x624b:0x0].0 bits 0x13 rrc: 5 type: IBT flags: 0x20 nid: 5822@gni100 remote: 0xa050f61ea309e3de expref: 125 pid: 15257 timeout: 0 lvb_type: 0 used 0 Jul 17 19:13:16 atlas-mds3.ccs.ornl.gov kernel: [ 5368.153148] Lustre: atlas2-MDT0000: Client 84eb7626-264a-9db5-eff3-ee82928c6490 (at 44@gni2) reconnecting, waiting for 20156 clients in recovery for 8:43 Jul 17 19:13:16 atlas-mds3.ccs.ornl.gov kernel: [ 5368.168735] Lustre: Skipped 372 previous similar messages Jul 17 19:13:16 atlas-mds3.ccs.ornl.gov kernel: [ 5368.174879] Lustre: atlas2-MDT0000: Client 84eb7626-264a-9db5-eff3-ee82928c6490 (at 44@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 19:13:16 atlas-mds3.ccs.ornl.gov kernel: [ 5368.189584] Lustre: Skipped 372 previous similar messages Jul 17 19:14:18 atlas-mds3.ccs.ornl.gov kernel: [ 5430.437601] Lustre: atlas2-MDT0000: Denying connection for new client 7c9cecb7-6c21-5cab-c1b6-5ab153ff6158 (at 8173@gni100), waiting for all 20156 known clients (18937 recovered, 1209 in progress, and 10 evicted) to recover in 7:41 Jul 17 19:14:18 atlas-mds3.ccs.ornl.gov kernel: [ 5430.462989] Lustre: Skipped 10 previous similar messages Jul 17 19:17:48 atlas-mds3.ccs.ornl.gov kernel: [ 5640.301063] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 565s: evicting client at 5822@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff881e3927e900/0xf35a0587ba95dbea lrc: 3/0,0 mode: PR/PR res: [0x2000d270e:0x624b:0x0].0 bits 0x13 rrc: 5 type: IBT flags: 0x20 nid: 5822@gni100 remote: 0xa050f61ea309e3de expref: 125 pid: 15257 timeout: 4300300143 lvb_type: 0 used 0 Jul 17 19:17:48 atlas-mds3.ccs.ornl.gov kernel: [ 5640.361671] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff883fce259d80/0xf35a0587ba96ebe7 lrc: 3/0,0 mode: PR/PR res: [0x2000d3d31:0xbd4:0x0].0 bits 0x13 rrc: 6 type: IBT flags: 0x20 nid: 13041@gni100 remote: 0x42ece4921752da28 expref: 129 pid: 16005 timeout: 0 lvb_type: 0 used 0 Jul 17 19:17:48 atlas-mds3.ccs.ornl.gov kernel: [ 5640.399835] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) Skipped 1 previous similar message Jul 17 19:23:16 atlas-mds3.ccs.ornl.gov kernel: [ 5968.733220] Lustre: atlas2-MDT0000: Client 84eb7626-264a-9db5-eff3-ee82928c6490 (at 44@gni2) reconnecting, waiting for 20156 clients in recovery for 0:-11 Jul 17 19:23:16 atlas-mds3.ccs.ornl.gov kernel: [ 5968.748903] Lustre: Skipped 372 previous similar messages Jul 17 19:23:16 atlas-mds3.ccs.ornl.gov kernel: [ 5968.755048] Lustre: atlas2-MDT0000: Client 84eb7626-264a-9db5-eff3-ee82928c6490 (at 44@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 19:23:16 atlas-mds3.ccs.ornl.gov kernel: [ 5968.769740] Lustre: Skipped 372 previous similar messages Jul 17 19:24:04 atlas-mds3.ccs.ornl.gov kernel: [ 6016.662215] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 13041@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff883fce259d80/0xf35a0587ba96ebe7 lrc: 3/0,0 mode: PR/PR res: [0x2000d3d31:0xbd4:0x0].0 bits 0x13 rrc: 6 type: IBT flags: 0x20 nid: 13041@gni100 remote: 0x42ece4921752da28 expref: 129 pid: 16005 timeout: 4300676110 lvb_type: 0 used 0 Jul 17 19:24:04 atlas-mds3.ccs.ornl.gov kernel: [ 6016.704075] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 1 previous similar message Jul 17 19:24:04 atlas-mds3.ccs.ornl.gov kernel: [ 6016.715619] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 19:25:31 atlas-mds3.ccs.ornl.gov kernel: [ 6104.065120] Lustre: atlas2-MDT0000: Denying connection for new client 1cbe0fe6-3a30-b20e-2504-2e3665d3e188 (at 11386@gni100), waiting for all 20156 known clients (18948 recovered, 1195 in progress, and 13 evicted) to recover in 3:49 Jul 17 19:25:31 atlas-mds3.ccs.ornl.gov kernel: [ 6104.088485] Lustre: Skipped 11 previous similar messages Jul 17 19:30:20 atlas-mds3.ccs.ornl.gov kernel: [ 6393.023255] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 19:30:20 atlas-mds3.ccs.ornl.gov kernel: [ 6393.205581] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff884004a0dd80/0xf35a0587ba95f09d lrc: 3/0,0 mode: PR/PR res: [0x2000d1f90:0x3164:0x0].0 bits 0x13 rrc: 199 type: IBT flags: 0x20 nid: 1105@gni100 remote: 0xef1db35c08978044 expref: 130 pid: 16223 timeout: 0 lvb_type: 0 used 0 Jul 17 19:30:20 atlas-mds3.ccs.ornl.gov kernel: [ 6393.243928] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) Skipped 1 previous similar message Jul 17 19:33:16 atlas-mds3.ccs.ornl.gov kernel: [ 6569.569328] Lustre: atlas2-MDT0000: Client fdf957bd-4457-0034-ecb5-687b2414db45 (at 31@gni2) reconnecting, waiting for 20156 clients in recovery for 2:20 Jul 17 19:33:16 atlas-mds3.ccs.ornl.gov kernel: [ 6569.584920] Lustre: Skipped 373 previous similar messages Jul 17 19:33:16 atlas-mds3.ccs.ornl.gov kernel: [ 6569.591071] Lustre: atlas2-MDT0000: Client fdf957bd-4457-0034-ecb5-687b2414db45 (at 31@gni2) refused reconnection, still busy with 19 active RPCs Jul 17 19:33:16 atlas-mds3.ccs.ornl.gov kernel: [ 6569.605870] Lustre: Skipped 373 previous similar messages Jul 17 19:36:36 atlas-mds3.ccs.ornl.gov kernel: [ 6769.384104] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 1105@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff884004a0dd80/0xf35a0587ba95f09d lrc: 3/0,0 mode: PR/PR res: [0x2000d1f90:0x3164:0x0].0 bits 0x13 rrc: 200 type: IBT flags: 0x20 nid: 1105@gni100 remote: 0xef1db35c08978044 expref: 130 pid: 16223 timeout: 4301428232 lvb_type: 0 used 0 Jul 17 19:36:36 atlas-mds3.ccs.ornl.gov kernel: [ 6769.426039] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 1 previous similar message Jul 17 19:36:36 atlas-mds3.ccs.ornl.gov kernel: [ 6769.450369] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 19:37:11 atlas-mds3.ccs.ornl.gov kernel: [ 6804.524295] Lustre: atlas2-MDT0000: Denying connection for new client 17f7a210-a88a-62e6-a6f9-583bd733e362 (at 10133@gni100), waiting for all 20156 known clients (19035 recovered, 1106 in progress, and 15 evicted) to recover in 4:42 Jul 17 19:37:11 atlas-mds3.ccs.ornl.gov kernel: [ 6804.547673] Lustre: Skipped 16 previous similar messages Jul 17 19:43:17 atlas-mds3.ccs.ornl.gov kernel: [ 7171.184620] Lustre: atlas2-MDT0000: Client 27762a33-afc6-af9d-7ab0-b0429d1d85e5 (at 16@gni2) reconnecting, waiting for 20156 clients in recovery for -1:-24 Jul 17 19:43:17 atlas-mds3.ccs.ornl.gov kernel: [ 7171.200408] Lustre: Skipped 371 previous similar messages Jul 17 19:43:17 atlas-mds3.ccs.ornl.gov kernel: [ 7171.200550] Lustre: atlas2-MDT0000: Client 1c0dca9e-5706-aca6-e344-f6aef05ad87b (at 14@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 19:43:17 atlas-mds3.ccs.ornl.gov kernel: [ 7171.200553] Lustre: Skipped 371 previous similar messages Jul 17 19:46:01 atlas-mds3.ccs.ornl.gov kernel: [ 7334.926321] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 19:46:01 atlas-mds3.ccs.ornl.gov kernel: [ 7334.990426] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff881d061c8480/0xf35a0587ba99324c lrc: 3/0,0 mode: PR/PR res: [0x2000d19f5:0xda0:0x0].0 bits 0x13 rrc: 6 type: IBT flags: 0x20 nid: 10000@gni100 remote: 0x60e19b123cfa2b59 expref: 130 pid: 15688 timeout: 0 lvb_type: 0 used 0 Jul 17 19:46:01 atlas-mds3.ccs.ornl.gov kernel: [ 7335.028623] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) Skipped 1 previous similar message Jul 17 19:47:20 atlas-mds3.ccs.ornl.gov kernel: [ 7414.263164] Lustre: atlas2-MDT0000: Denying connection for new client 81a83a10-01e6-f065-3b93-de8830face64 (at 7131@gni100), waiting for all 20156 known clients (19059 recovered, 1081 in progress, and 16 evicted) to recover in 3:57 Jul 17 19:47:20 atlas-mds3.ccs.ornl.gov kernel: [ 7414.286445] Lustre: Skipped 17 previous similar messages Jul 17 19:53:17 atlas-mds3.ccs.ornl.gov kernel: [ 7771.761192] Lustre: atlas2-MDT0000: Client 27762a33-afc6-af9d-7ab0-b0429d1d85e5 (at 16@gni2) reconnecting, waiting for 20156 clients in recovery for -1:-59 Jul 17 19:53:17 atlas-mds3.ccs.ornl.gov kernel: [ 7771.777063] Lustre: Skipped 372 previous similar messages Jul 17 19:53:17 atlas-mds3.ccs.ornl.gov kernel: [ 7771.783218] Lustre: atlas2-MDT0000: Client 27762a33-afc6-af9d-7ab0-b0429d1d85e5 (at 16@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 19:53:17 atlas-mds3.ccs.ornl.gov kernel: [ 7771.797929] Lustre: Skipped 373 previous similar messages Jul 17 19:56:02 atlas-mds3.ccs.ornl.gov kernel: [ 7936.501772] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 601s: evicting client at 10000@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff881d061c8480/0xf35a0587ba99324c lrc: 3/0,0 mode: PR/PR res: [0x2000d19f5:0xda0:0x0].0 bits 0x13 rrc: 6 type: IBT flags: 0x20 nid: 10000@gni100 remote: 0x60e19b123cfa2b59 expref: 129 pid: 15688 timeout: 4302594114 lvb_type: 0 used 0 Jul 17 19:56:02 atlas-mds3.ccs.ornl.gov kernel: [ 7936.543614] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 1 previous similar message Jul 17 19:56:02 atlas-mds3.ccs.ornl.gov kernel: [ 7936.555236] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 19:56:02 atlas-mds3.ccs.ornl.gov kernel: [ 7936.624208] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff8840070c7240/0xf35a0587ba980ead lrc: 3/0,0 mode: PR/PR res: [0x2000d2dbe:0x1bd9:0x0].0 bits 0x13 rrc: 13 type: IBT flags: 0x20 nid: 10122@gni100 remote: 0x5129e15bf3ad29a1 expref: 128 pid: 16030 timeout: 0 lvb_type: 0 used 0 Jul 17 19:58:53 atlas-mds3.ccs.ornl.gov kernel: [ 8107.899082] Lustre: atlas2-MDT0000: Denying connection for new client e7e497f0-c155-31e1-a97b-4cbe4568679c (at 2760@gni100), waiting for all 20156 known clients (19081 recovered, 1058 in progress, and 17 evicted) to recover in 2:25 Jul 17 19:58:53 atlas-mds3.ccs.ornl.gov kernel: [ 8107.922363] Lustre: Skipped 17 previous similar messages Jul 17 20:02:18 atlas-mds3.ccs.ornl.gov kernel: [ 8312.862421] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 20:02:18 atlas-mds3.ccs.ornl.gov kernel: [ 8312.872484] Lustre: atlas2-MDT0000: disconnecting 1 stale clients Jul 17 20:03:17 atlas-mds3.ccs.ornl.gov kernel: [ 8372.347166] Lustre: atlas2-MDT0000: Client 27762a33-afc6-af9d-7ab0-b0429d1d85e5 (at 16@gni2) reconnecting, waiting for 20156 clients in recovery for 4:17 Jul 17 20:03:17 atlas-mds3.ccs.ornl.gov kernel: [ 8372.362775] Lustre: Skipped 372 previous similar messages Jul 17 20:03:17 atlas-mds3.ccs.ornl.gov kernel: [ 8372.368933] Lustre: atlas2-MDT0000: Client 27762a33-afc6-af9d-7ab0-b0429d1d85e5 (at 16@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 20:03:17 atlas-mds3.ccs.ornl.gov kernel: [ 8372.383653] Lustre: Skipped 372 previous similar messages Jul 17 20:08:34 atlas-mds3.ccs.ornl.gov kernel: [ 8689.221782] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 9730@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff883fc92e7480/0xf35a0587ba997ee3 lrc: 3/0,0 mode: PR/PR res: [0x2000d2f9a:0xc06:0x0].0 bits 0x13 rrc: 9 type: IBT flags: 0x20 nid: 9730@gni100 remote: 0x5ccefb4b2822092d expref: 129 pid: 15891 timeout: 4303346027 lvb_type: 0 used 0 Jul 17 20:08:34 atlas-mds3.ccs.ornl.gov kernel: [ 8689.263439] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 1 previous similar message Jul 17 20:08:34 atlas-mds3.ccs.ornl.gov kernel: [ 8689.287390] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 20:08:34 atlas-mds3.ccs.ornl.gov kernel: [ 8689.325886] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff881d06e30900/0xf35a0587ba982321 lrc: 3/0,0 mode: PR/PR res: [0x2000d1f90:0x316c:0x0].0 bits 0x13 rrc: 131 type: IBT flags: 0x20 nid: 10516@gni100 remote: 0xa01bd615afb08d78 expref: 127 pid: 15550 timeout: 0 lvb_type: 0 used 0 Jul 17 20:08:34 atlas-mds3.ccs.ornl.gov kernel: [ 8689.364338] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) Skipped 2 previous similar messages Jul 17 20:09:24 atlas-mds3.ccs.ornl.gov kernel: [ 8739.604058] Lustre: atlas2-MDT0000: Denying connection for new client 7c9cecb7-6c21-5cab-c1b6-5ab153ff6158 (at 8173@gni100), waiting for all 20156 known clients (19103 recovered, 1032 in progress, and 21 evicted) to recover in 4:26 Jul 17 20:09:24 atlas-mds3.ccs.ornl.gov kernel: [ 8739.627327] Lustre: Skipped 18 previous similar messages Jul 17 20:12:20 atlas-oss1f1.ccs.ornl.gov kernel: [791851.325148] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 13@gni110 ns: filter-atlas1-OST01d8_UUID lock: ffff88021f7b0d80/0x11123c2f0d43ee68 lrc: 3/0,0 mode: PW/PW res: [0x570fa6:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80010020 nid: 13@gni110 remote: 0xb5b5113aa269dba2 expref: 12 pid: 38686 timeout: 5086218604 lvb_type: 0 Jul 17 20:13:18 atlas-mds3.ccs.ornl.gov kernel: [ 8973.664327] Lustre: atlas2-MDT0000: Client 1d186399-bf63-fee6-2b63-8fddd9e7fba3 (at 83@gni2) reconnecting, waiting for 20156 clients in recovery for 0:32 Jul 17 20:13:18 atlas-mds3.ccs.ornl.gov kernel: [ 8973.679915] Lustre: Skipped 374 previous similar messages Jul 17 20:13:18 atlas-mds3.ccs.ornl.gov kernel: [ 8973.685115] Lustre: atlas2-MDT0000: Client f27002de-ab6c-80a4-91a7-0d2824607322 (at 80@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 20:13:18 atlas-mds3.ccs.ornl.gov kernel: [ 8973.685117] Lustre: Skipped 374 previous similar messages Jul 17 20:14:50 atlas-mds3.ccs.ornl.gov kernel: [ 9065.583029] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 20:20:37 atlas-mds3.ccs.ornl.gov kernel: [ 9413.230945] Lustre: atlas2-MDT0000: Denying connection for new client 1cbe0fe6-3a30-b20e-2504-2e3665d3e188 (at 11386@gni100), waiting for all 20156 known clients (19126 recovered, 1008 in progress, and 22 evicted) to recover in 0:-30 Jul 17 20:20:37 atlas-mds3.ccs.ornl.gov kernel: [ 9413.254413] Lustre: Skipped 24 previous similar messages Jul 17 20:21:06 atlas-mds3.ccs.ornl.gov kernel: [ 9441.943231] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 11681@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff881d07c7f240/0xf35a0587ba970762 lrc: 3/0,0 mode: PR/PR res: [0x2000d1992:0x47f7:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x20 nid: 11681@gni100 remote: 0x52f7294368d0e7c4 expref: 129 pid: 15529 timeout: 4304098076 lvb_type: 0 used 0 Jul 17 20:21:06 atlas-mds3.ccs.ornl.gov kernel: [ 9441.985270] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 2 previous similar messages Jul 17 20:21:06 atlas-mds3.ccs.ornl.gov kernel: [ 9442.001611] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 20:21:06 atlas-mds3.ccs.ornl.gov kernel: [ 9442.024632] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff883fc7b21240/0xf35a0587ba988598 lrc: 3/0,0 mode: PR/PR res: [0x2000d44ff:0x2e4e:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x20 nid: 4287@gni100 remote: 0xf8c30106a88ac626 expref: 131 pid: 15854 timeout: 0 lvb_type: 0 used 0 Jul 17 20:21:06 atlas-mds3.ccs.ornl.gov kernel: [ 9442.062793] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) Skipped 1 previous similar message Jul 17 20:22:10 atlas-mds1.ccs.ornl.gov kernel: [807709.462300] LustreError: 26795:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: mdt-atlas1-MDT0000_UUID lock: ffff882edb5e7240/0x52a25574c401158f lrc: 3/0,0 mode: PR/PR res: [0x20008e2ff:0xc2cd:0x0].0 bits 0x13 rrc: 5 type: IBT flags: 0x200000000020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac8c8b3d expref: 236281 pid: 27179 timeout: 5101603190 lvb_type: 0 used 0 Jul 17 20:23:18 atlas-mds3.ccs.ornl.gov kernel: [ 9574.246608] Lustre: atlas2-MDT0000: Client f27002de-ab6c-80a4-91a7-0d2824607322 (at 80@gni2) reconnecting, waiting for 20156 clients in recovery for 3:04 Jul 17 20:23:18 atlas-mds3.ccs.ornl.gov kernel: [ 9574.262195] Lustre: Skipped 368 previous similar messages Jul 17 20:23:18 atlas-mds3.ccs.ornl.gov kernel: [ 9574.268355] Lustre: atlas2-MDT0000: Client f27002de-ab6c-80a4-91a7-0d2824607322 (at 80@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 20:23:18 atlas-mds3.ccs.ornl.gov kernel: [ 9574.283062] Lustre: Skipped 368 previous similar messages Jul 17 20:27:22 atlas-mds3.ccs.ornl.gov kernel: [ 9818.311480] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 20:28:26 atlas-oss1d2.ccs.ornl.gov kernel: [792816.259425] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0379_UUID lock: ffff880ad1f64480/0x80696b938b4b404d lrc: 3/0,0 mode: PW/PW res: [0x56039f:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96efec expref: 756 pid: 38470 timeout: 5087182993 lvb_type: 0 Jul 17 20:28:26 atlas-oss1d2.ccs.ornl.gov kernel: [792816.384548] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0139_UUID lock: ffff880a0650eb40/0x80696b938b4b412d lrc: 3/0,0 mode: PW/PW res: [0x5c6f65:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f04e expref: 737 pid: 38642 timeout: 5087182993 lvb_type: 0 Jul 17 20:28:26 atlas-oss2e2.ccs.ornl.gov kernel: [785564.194512] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST02a9_UUID lock: ffff8809f5ee2000/0xf2a2d9b42c6cb632 lrc: 3/0,0 mode: PW/PW res: [0x5388a6:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96efe5 expref: 625 pid: 118695 timeout: 5079937986 lvb_type: 0 Jul 17 20:28:26 atlas-oss2f4.ccs.ornl.gov kernel: [785565.684851] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0193_UUID lock: ffff880a89177d80/0x1a980fbd0eb536e5 lrc: 3/0,0 mode: PW/PW res: [0x4f97ca:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fa50 expref: 580 pid: 118091 timeout: 5079937990 lvb_type: 0 Jul 17 20:28:26 atlas-oss1a2.ccs.ornl.gov kernel: [792814.545215] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0001_UUID lock: ffff880dc2d80240/0x4f5405432a3fe275 lrc: 3/0,0 mode: PW/PW res: [0x528db2:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f303 expref: 642 pid: 39890 timeout: 5087183980 lvb_type: 0 Jul 17 20:28:26 atlas-oss1a2.ccs.ornl.gov kernel: [792814.671940] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0121_UUID lock: ffff880e91e46b40/0x4f5405432a40153c lrc: 3/0,0 mode: PW/PW res: [0x58c2d6:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fcfe expref: 712 pid: 40025 timeout: 5087183982 lvb_type: 0 Jul 17 20:28:26 atlas-oss2e8.ccs.ornl.gov kernel: [785567.320319] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST00ff_UUID lock: ffff880f84469d80/0x1d4ea6f890b1266c lrc: 3/0,0 mode: PW/PW res: [0x588980:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fc9c expref: 703 pid: 117871 timeout: 5079934975 lvb_type: 0 Jul 17 20:28:26 atlas-oss1e5.ccs.ornl.gov kernel: [792812.394179] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST01d4_UUID lock: ffff880c8107fd80/0x335df11698069a0d lrc: 3/0,0 mode: PW/PW res: [0x52c650:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f127 expref: 677 pid: 38099 timeout: 5087183959 lvb_type: 0 Jul 17 20:28:26 atlas-oss1i4.ccs.ornl.gov kernel: [792814.233620] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0283_UUID lock: ffff880975e57b40/0xfc6d4e6fa254e30d lrc: 3/0,0 mode: PW/PW res: [0x583218:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f182 expref: 672 pid: 38722 timeout: 5087185954 lvb_type: 0 Jul 17 20:28:26 atlas-oss1i4.ccs.ornl.gov kernel: [792814.358176] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST00d3_UUID lock: ffff880e0d9ae6c0/0xfc6d4e6fa254ec8a lrc: 3/0,0 mode: PW/PW res: [0x53e3c5:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f484 expref: 647 pid: 38617 timeout: 5087185955 lvb_type: 0 Jul 17 20:28:26 atlas-oss2i7.ccs.ornl.gov kernel: [785559.495561] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST03ee_UUID lock: ffff880fe016c900/0x9090eb45c43d6299 lrc: 3/0,0 mode: PW/PW res: [0x5cd130:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f429 expref: 778 pid: 118515 timeout: 5079937955 lvb_type: 0 Jul 17 20:28:26 atlas-oss1a6.ccs.ornl.gov kernel: [792818.116740] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0005_UUID lock: ffff8808d0090240/0xe0122325cb760b9f lrc: 3/0,0 mode: PW/PW res: [0x52d404:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f189 expref: 641 pid: 38568 timeout: 5087182929 lvb_type: 0 Jul 17 20:28:26 atlas-oss2a4.ccs.ornl.gov kernel: [785566.173819] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST028b_UUID lock: ffff880decf3eb40/0x31a79a96c8cf9583 lrc: 3/0,0 mode: PW/PW res: [0x5476fc:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f739 expref: 638 pid: 117472 timeout: 5079937921 lvb_type: 0 Jul 17 20:28:26 atlas-oss2f2.ccs.ornl.gov kernel: [785561.827107] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0221_UUID lock: ffff880a3350f000/0x59f6e801da891f3a lrc: 3/0,0 mode: PW/PW res: [0x519014:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f238 expref: 646 pid: 117642 timeout: 5079934863 lvb_type: 0 Jul 17 20:28:26 atlas-oss2c8.ccs.ornl.gov kernel: [785566.949237] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST020f_UUID lock: ffff880963759480/0x9a947a799d0bfaba lrc: 3/0,0 mode: PW/PW res: [0x59d767:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f9e0 expref: 738 pid: 1010 timeout: 5079934868 lvb_type: 0 Jul 17 20:28:26 atlas-oss2i1.ccs.ornl.gov kernel: [784846.936378] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0118_UUID lock: ffff880f9aff1d80/0xf87501f8ab9eb5f6 lrc: 3/0,0 mode: PW/PW res: [0x511441:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fbca expref: 640 pid: 114496 timeout: 5079225839 lvb_type: 0 Jul 17 20:28:26 atlas-oss2e5.ccs.ornl.gov kernel: [785565.425388] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST018c_UUID lock: ffff880ad4c6b6c0/0x58361805d207160a lrc: 3/0,0 mode: PW/PW res: [0x57fa5e:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fbd1 expref: 687 pid: 118247 timeout: 5079936838 lvb_type: 0 Jul 17 20:28:26 atlas-oss1c2.ccs.ornl.gov kernel: [792814.408538] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0251_UUID lock: ffff880ceb807900/0xaf4617cc9039f6f8 lrc: 3/0,0 mode: PW/PW res: [0x526291:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f923 expref: 642 pid: 38103 timeout: 5087185823 lvb_type: 0 Jul 17 20:28:26 atlas-oss2f1.ccs.ornl.gov kernel: [785514.230938] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0190_UUID lock: ffff880c1424d900/0xa162e314dafadcb lrc: 3/0,0 mode: PW/PW res: [0x4ee8a7:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f499 expref: 614 pid: 113620 timeout: 5079882823 lvb_type: 0 Jul 17 20:28:26 atlas-oss2f1.ccs.ornl.gov kernel: [785514.358955] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0220_UUID lock: ffff881006d18d80/0xa162e314dafd111 lrc: 3/0,0 mode: PW/PW res: [0x5403d9:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fa49 expref: 656 pid: 113620 timeout: 5079882825 lvb_type: 0 Jul 17 20:28:26 atlas-oss1g5.ccs.ornl.gov kernel: [712709.239855] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0304_UUID lock: ffff880cc0d77480/0x40c6faaa64f8705f lrc: 3/0,0 mode: PW/PW res: [0x4c72b9:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f1dd expref: 586 pid: 18160 timeout: 5007106818 lvb_type: 0 Jul 17 20:28:26 atlas-oss1g5.ccs.ornl.gov kernel: [712709.364343] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0394_UUID lock: ffff880a7d824480/0x40c6faaa64f87242 lrc: 3/0,0 mode: PW/PW res: [0x4f2bd6:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f2a1 expref: 613 pid: 16880 timeout: 5007106818 lvb_type: 0 Jul 17 20:28:26 atlas-oss1d5.ccs.ornl.gov kernel: [792817.878946] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST037c_UUID lock: ffff880fe253ab40/0x8dbaed84346d160d lrc: 3/0,0 mode: PW/PW res: [0x543726:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f120 expref: 637 pid: 39889 timeout: 5087185784 lvb_type: 0 Jul 17 20:28:26 atlas-oss2h4.ccs.ornl.gov kernel: [785563.049912] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0113_UUID lock: ffff88090cf79240/0x1b817f7c90dd2d59 lrc: 3/0,0 mode: PW/PW res: [0x53356e:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f5b1 expref: 677 pid: 118053 timeout: 5079937790 lvb_type: 0 Jul 17 20:28:26 atlas-oss2h4.ccs.ornl.gov kernel: [785563.181966] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST03e3_UUID lock: ffff880b81293000/0x1b817f7c90dd4d8f lrc: 3/0,0 mode: PW/PW res: [0x58dd5e:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fb7d expref: 707 pid: 118262 timeout: 5079937791 lvb_type: 0 Jul 17 20:28:26 atlas-oss2a2.ccs.ornl.gov kernel: [785564.587152] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0319_UUID lock: ffff880a888b3b40/0xdf98a27155d08c83 lrc: 3/0,0 mode: PW/PW res: [0x5888ba:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96effa expref: 700 pid: 118333 timeout: 5079936779 lvb_type: 0 Jul 17 20:28:26 atlas-oss2a2.ccs.ornl.gov kernel: [785564.715666] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0319_UUID lock: ffff880c785f4b40/0xdf98a27155d0abb6 lrc: 3/0,0 mode: PW/PW res: [0x588905:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f8c8 expref: 700 pid: 118333 timeout: 5079936782 lvb_type: 0 Jul 17 20:28:26 atlas-oss1i8.ccs.ornl.gov kernel: [792816.336621] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0167_UUID lock: ffff880bff2d5240/0xc3c7b4898654a8a2 lrc: 3/0,0 mode: PW/PW res: [0x56667a:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f3d5 expref: 684 pid: 38697 timeout: 5087183778 lvb_type: 0 Jul 17 20:28:26 atlas-oss1b6.ccs.ornl.gov kernel: [792814.816658] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST024d_UUID lock: ffff880d85aa0b40/0x2bb6c00daef4f000 lrc: 3/0,0 mode: PW/PW res: [0x507c0f:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f66e expref: 572 pid: 94090 timeout: 5087185775 lvb_type: 0 Jul 17 20:28:26 atlas-oss1b6.ccs.ornl.gov kernel: [792814.946681] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST036d_UUID lock: ffff880b0d3c56c0/0x2bb6c00daef50cbd lrc: 3/0,0 mode: PW/PW res: [0x56a53c:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fab9 expref: 692 pid: 94090 timeout: 5087185776 lvb_type: 0 Jul 17 20:28:26 atlas-oss2c6.ccs.ornl.gov kernel: [785559.984670] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST00ed_UUID lock: ffff880ed0e40240/0x6dcd3154f561fe28 lrc: 3/0,0 mode: PW/PW res: [0x559a05:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f91c expref: 690 pid: 122830 timeout: 5079937864 lvb_type: 0 Jul 17 20:28:26 atlas-oss2c6.ccs.ornl.gov kernel: [785560.115236] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST005d_UUID lock: ffff880eb42b8000/0x6dcd3154f5620a14 lrc: 3/0,0 mode: PW/PW res: [0x5923db:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fc2c expref: 723 pid: 122830 timeout: 5079937865 lvb_type: 0 Jul 17 20:28:26 atlas-oss1d4.ccs.ornl.gov kernel: [792814.242643] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST037b_UUID lock: ffff8808b3595900/0xb6190dbaa664e6 lrc: 3/0,0 mode: PW/PW res: [0x52ecac:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f67c expref: 643 pid: 38781 timeout: 5087183744 lvb_type: 0 Jul 17 20:28:26 atlas-oss1d4.ccs.ornl.gov kernel: [792814.373244] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST001b_UUID lock: ffff8809c4c9ab40/0xb6190dbaa664ed lrc: 3/0,0 mode: PW/PW res: [0x558d0b:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f683 expref: 691 pid: 38781 timeout: 5087183744 lvb_type: 0 Jul 17 20:28:26 atlas-oss1d6.ccs.ornl.gov kernel: [792819.715585] LustreError: 25341:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST013d_UUID lock: ffff880a255a9000/0xe23be95afc1bd7fe lrc: 3/0,0 mode: PW/PW res: [0x5349a1:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f3ce expref: 697 pid: 38889 timeout: 5087185741 lvb_type: 0 Jul 17 20:28:26 atlas-oss1h1.ccs.ornl.gov kernel: [792816.609656] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0308_UUID lock: ffff880c09bb5480/0x8e5a47084bba9a4a lrc: 3/0,0 mode: PW/PW res: [0x537078:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f1eb expref: 628 pid: 38916 timeout: 5087182705 lvb_type: 0 Jul 17 20:28:26 atlas-oss2b4.ccs.ornl.gov kernel: [785562.506762] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0323_UUID lock: ffff880d0a1a8900/0xb6b2b62271350280 lrc: 3/0,0 mode: PW/PW res: [0x52f8bd:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fc8e expref: 614 pid: 114952 timeout: 5079937695 lvb_type: 0 Jul 17 20:28:26 atlas-oss1f4.ccs.ornl.gov kernel: [792817.387858] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST01db_UUID lock: ffff880a84a6db40/0xc53d45d0f6895e0f lrc: 3/0,0 mode: PW/PW res: [0x573779:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f24d expref: 705 pid: 39357 timeout: 5087185686 lvb_type: 0 Jul 17 20:28:26 atlas-oss1d7.ccs.ornl.gov kernel: [792818.694472] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST025e_UUID lock: ffff8808f5329480/0xfa14491a02801377 lrc: 3/0,0 mode: PW/PW res: [0x54d83e:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f35e expref: 675 pid: 38639 timeout: 5087185816 lvb_type: 0 Jul 17 20:28:26 atlas-oss1d7.ccs.ornl.gov kernel: [792818.822705] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST01ce_UUID lock: ffff880cdc2f8900/0xfa14491a02802edd lrc: 3/0,0 mode: PW/PW res: [0x530eef:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f858 expref: 590 pid: 38731 timeout: 5087185818 lvb_type: 0 Jul 17 20:28:26 atlas-oss2a3.ccs.ornl.gov kernel: [785568.688504] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST016a_UUID lock: ffff8809df130480/0x70099c07d52742a9 lrc: 3/0,0 mode: PW/PW res: [0x5589fd:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f6de expref: 673 pid: 25192 timeout: 5079936669 lvb_type: 0 Jul 17 20:28:26 atlas-oss1i2.ccs.ornl.gov kernel: [792814.797036] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0161_UUID lock: ffff880ed170ab40/0x8d72e7ac16470a61 lrc: 3/0,0 mode: PW/PW res: [0x53cc57:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f0b0 expref: 634 pid: 38412 timeout: 5087182666 lvb_type: 0 Jul 17 20:28:26 atlas-oss1g1.ccs.ornl.gov kernel: [712705.222278] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0300_UUID lock: ffff880c407ed900/0xd7bb41931e20be99 lrc: 3/0,0 mode: PW/PW res: [0x56e2eb:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f85f expref: 669 pid: 19790 timeout: 5007101654 lvb_type: 0 Jul 17 20:28:26 atlas-oss1i5.ccs.ornl.gov kernel: [792812.247500] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0284_UUID lock: ffff880fa5fc7d80/0x4fceeff5af70e955 lrc: 3/0,0 mode: PW/PW res: [0x553809:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f4f4 expref: 670 pid: 38855 timeout: 5087182651 lvb_type: 0 Jul 17 20:28:26 atlas-oss2d1.ccs.ornl.gov kernel: [785561.307771] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0060_UUID lock: ffff880d8c21d900/0x8acfe5f0740a3b83 lrc: 3/0,0 mode: PW/PW res: [0x55c2bb:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f675 expref: 668 pid: 118379 timeout: 5079936630 lvb_type: 0 Jul 17 20:28:26 atlas-oss2h8.ccs.ornl.gov kernel: [784850.717307] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0087_UUID lock: ffff880de3fe46c0/0x763e041f475bf6e1 lrc: 3/0,0 mode: PW/PW res: [0x5c61f4:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96eff3 expref: 766 pid: 114299 timeout: 5079229752 lvb_type: 0 Jul 17 20:28:26 atlas-oss2h8.ccs.ornl.gov kernel: [784850.846623] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0357_UUID lock: ffff880970fee240/0x763e041f475c091e lrc: 3/0,0 mode: PW/PW res: [0x56a2fd:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f492 expref: 677 pid: 21297 timeout: 5079229754 lvb_type: 0 Jul 17 20:28:26 atlas-oss1f2.ccs.ornl.gov kernel: [792818.020449] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST01d9_UUID lock: ffff880eaa3d9240/0xef50898f7aa47336 lrc: 3/0,0 mode: PW/PW res: [0x58d38e:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f621 expref: 706 pid: 38938 timeout: 5087183622 lvb_type: 0 Jul 17 20:28:26 atlas-oss1f6.ccs.ornl.gov kernel: [792763.131450] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST002d_UUID lock: ffff880bdfbb2000/0xe15c7d03ca4daafc lrc: 3/0,0 mode: PW/PW res: [0x578e25:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96faa4 expref: 768 pid: 38482 timeout: 5087130599 lvb_type: 0 Jul 17 20:28:26 atlas-oss2h2.ccs.ornl.gov kernel: [785566.611183] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0081_UUID lock: ffff880f489c7b40/0xbc38718aa4709874 lrc: 3/0,0 mode: PW/PW res: [0x581b60:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f047 expref: 742 pid: 118134 timeout: 5079936598 lvb_type: 0 Jul 17 20:28:26 atlas-oss2h2.ccs.ornl.gov kernel: [785566.744349] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0111_UUID lock: ffff880c0ed516c0/0xbc38718aa470a746 lrc: 3/0,0 mode: PW/PW res: [0x563443:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f54f expref: 666 pid: 118176 timeout: 5079936600 lvb_type: 0 Jul 17 20:28:26 atlas-oss2d7.ccs.ornl.gov kernel: [785567.853782] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST03c6_UUID lock: ffff880e2da46d80/0x102513dc75cecddd lrc: 3/0,0 mode: PW/PW res: [0x5f3193:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f92a expref: 779 pid: 116998 timeout: 5079936579 lvb_type: 0 Jul 17 20:28:26 atlas-oss1g2.ccs.ornl.gov kernel: [712705.387499] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST01e1_UUID lock: ffff880b39ef0000/0xe423ec7becdc2807 lrc: 3/0,0 mode: PW/PW res: [0x539bd0:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fc3a expref: 670 pid: 16582 timeout: 5007101537 lvb_type: 0 Jul 17 20:28:26 atlas-oss2d4.ccs.ornl.gov kernel: [785510.835793] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST03c3_UUID lock: ffff880d21dfc480/0xd8d6ae7c299ca43d lrc: 3/0,0 mode: PW/PW res: [0x569dc3:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f55d expref: 676 pid: 117606 timeout: 5079879510 lvb_type: 0 Jul 17 20:28:26 atlas-oss2d4.ccs.ornl.gov kernel: [785510.960553] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0333_UUID lock: ffff880c705316c0/0xd8d6ae7c299ca690 lrc: 3/0,0 mode: PW/PW res: [0x5695c3:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f5bf expref: 668 pid: 117606 timeout: 5079879511 lvb_type: 0 Jul 17 20:28:26 atlas-oss1a4.ccs.ornl.gov kernel: [792806.730612] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST02d3_UUID lock: ffff880d2abeb000/0xbf3dfea674414e07 lrc: 3/0,0 mode: PW/PW res: [0x535996:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fcf7 expref: 678 pid: 94180 timeout: 5087182497 lvb_type: 0 Jul 17 20:28:26 atlas-oss1a7.ccs.ornl.gov kernel: [792812.502018] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0006_UUID lock: ffff880fb0411d80/0xfead5526b9d75978 lrc: 3/0,0 mode: PW/PW res: [0x52245d:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f7a9 expref: 612 pid: 38916 timeout: 5087184494 lvb_type: 0 Jul 17 20:28:26 atlas-oss1a7.ccs.ornl.gov kernel: [792812.635078] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0096_UUID lock: ffff8808d757a000/0xfead5526b9d77428 lrc: 3/0,0 mode: PW/PW res: [0x5a7498:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fd05 expref: 733 pid: 38760 timeout: 5087184495 lvb_type: 0 Jul 17 20:28:26 atlas-oss1a5.ccs.ornl.gov kernel: [792061.200722] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0364_UUID lock: ffff880fb21ab480/0x2a153e09dbbefc54 lrc: 3/0,0 mode: PW/PW res: [0x58e214:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f1d6 expref: 683 pid: 34520 timeout: 5086430484 lvb_type: 0 Jul 17 20:28:26 atlas-oss1a5.ccs.ornl.gov kernel: [792061.327325] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0244_UUID lock: ffff880cb4918240/0x2a153e09dbbf33ac lrc: 3/0,0 mode: PW/PW res: [0x5c6e5b:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fb76 expref: 753 pid: 34430 timeout: 5086430487 lvb_type: 0 Jul 17 20:28:26 atlas-oss1b4.ccs.ornl.gov kernel: [792817.974822] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST012b_UUID lock: ffff880dd25a5480/0xd18237f829dfb146 lrc: 3/0,0 mode: PW/PW res: [0x5a09ae:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f3c7 expref: 700 pid: 38177 timeout: 5087184480 lvb_type: 0 Jul 17 20:28:26 atlas-oss1a3.ccs.ornl.gov kernel: [792813.610319] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0122_UUID lock: ffff880cdd8c1000/0x415b8255ba22db45 lrc: 3/0,0 mode: PW/PW res: [0x574b6d:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fb0d expref: 703 pid: 38559 timeout: 5087182481 lvb_type: 0 Jul 17 20:28:26 atlas-oss1a3.ccs.ornl.gov kernel: [792813.735180] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0092_UUID lock: ffff880cdd8c1480/0x415b8255ba22dea2 lrc: 3/0,0 mode: PW/PW res: [0x548002:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fc33 expref: 676 pid: 38756 timeout: 5087182481 lvb_type: 0 Jul 17 20:28:26 atlas-oss2f7.ccs.ornl.gov kernel: [785566.792219] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0226_UUID lock: ffff880929ae9900/0x2a3a728cf693410f lrc: 3/0,0 mode: PW/PW res: [0x5648bb:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f556 expref: 669 pid: 23836 timeout: 5079937480 lvb_type: 0 Jul 17 20:28:26 atlas-oss1c8.ccs.ornl.gov kernel: [792816.936271] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0257_UUID lock: ffff880d61778000/0x2ffe4f2f2042569 lrc: 3/0,0 mode: PW/PW res: [0x564531:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f8ba expref: 671 pid: 38825 timeout: 5087184480 lvb_type: 0 Jul 17 20:28:26 atlas-oss1c8.ccs.ornl.gov kernel: [792817.063502] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0257_UUID lock: ffff880edbc1bb40/0x2ffe4f2f2043434 lrc: 3/0,0 mode: PW/PW res: [0x56454a:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fb1b expref: 671 pid: 38844 timeout: 5087184480 lvb_type: 0 Jul 17 20:28:26 atlas-oss1i1.ccs.ornl.gov kernel: [792817.688818] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0310_UUID lock: ffff880bf872b6c0/0x372db933943bef74 lrc: 3/0,0 mode: PW/PW res: [0x560897:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f174 expref: 693 pid: 38903 timeout: 5087184472 lvb_type: 0 Jul 17 20:28:26 atlas-oss1i1.ccs.ornl.gov kernel: [792817.814150] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0280_UUID lock: ffff8808ae3c9b40/0x372db933943bf873 lrc: 3/0,0 mode: PW/PW res: [0x5304dc:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f437 expref: 629 pid: 38903 timeout: 5087184473 lvb_type: 0 Jul 17 20:28:26 atlas-oss2e1.ccs.ornl.gov kernel: [785569.958108] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST02a8_UUID lock: ffff880bfedc4900/0xc1c3dc5bff82af66 lrc: 3/0,0 mode: PW/PW res: [0x5208ae:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f5b8 expref: 613 pid: 118432 timeout: 5079938472 lvb_type: 0 Jul 17 20:28:26 atlas-oss2g6.ccs.ornl.gov kernel: [785562.292749] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST010d_UUID lock: ffff880eda034d80/0x5e7f571017bc07f6 lrc: 3/0,0 mode: PW/PW res: [0x541b69:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f29a expref: 657 pid: 114005 timeout: 5079936469 lvb_type: 0 Jul 17 20:28:26 atlas-oss2g6.ccs.ornl.gov kernel: [785562.417716] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST022d_UUID lock: ffff880959cce000/0x5e7f571017bc1d66 lrc: 3/0,0 mode: PW/PW res: [0x554139:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f7fd expref: 711 pid: 114007 timeout: 5079936471 lvb_type: 0 Jul 17 20:28:26 atlas-oss1b7.ccs.ornl.gov kernel: [792811.492981] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST012e_UUID lock: ffff880db8404000/0xd8d9cd61c17235fc lrc: 3/0,0 mode: PW/PW res: [0x57d2e7:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f0c5 expref: 726 pid: 38837 timeout: 5087184469 lvb_type: 0 Jul 17 20:28:26 atlas-oss1b7.ccs.ornl.gov kernel: [792811.620168] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST024e_UUID lock: ffff880ce7a48d80/0xd8d9cd61c1723c46 lrc: 3/0,0 mode: PW/PW res: [0x5271f5:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f373 expref: 657 pid: 38921 timeout: 5087184470 lvb_type: 0 Jul 17 20:28:26 atlas-oss1e8.ccs.ornl.gov kernel: [792817.179015] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0027_UUID lock: ffff880eb3a656c0/0x24931d398ab34d1e lrc: 3/0,0 mode: PW/PW res: [0x55c678:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f0be expref: 629 pid: 37498 timeout: 5087184473 lvb_type: 0 Jul 17 20:28:26 atlas-oss2h3.ccs.ornl.gov kernel: [785567.231583] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST02c2_UUID lock: ffff880c7c737480/0x14f94eeb9a7a396a lrc: 3/0,0 mode: PW/PW res: [0x54cff7:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f30a expref: 656 pid: 118034 timeout: 5079936474 lvb_type: 0 Jul 17 20:28:26 atlas-oss2h3.ccs.ornl.gov kernel: [785567.364049] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST03e2_UUID lock: ffff880cf90036c0/0x14f94eeb9a7a3971 lrc: 3/0,0 mode: PW/PW res: [0x52c7c6:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f311 expref: 679 pid: 118034 timeout: 5079936474 lvb_type: 0 Jul 17 20:28:26 atlas-oss1e7.ccs.ornl.gov kernel: [792817.389792] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0026_UUID lock: ffff880f4dd71480/0xa0eabd2702948a92 lrc: 3/0,0 mode: PW/PW res: [0x518150:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f422 expref: 618 pid: 37449 timeout: 5087184474 lvb_type: 0 Jul 17 20:28:26 atlas-oss2d8.ccs.ornl.gov kernel: [785564.271010] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0337_UUID lock: ffff8808b86d5900/0xf564dec7cde4e4b6 lrc: 3/0,0 mode: PW/PW res: [0x56b051:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f8cf expref: 673 pid: 118225 timeout: 5079935472 lvb_type: 0 Jul 17 20:28:26 atlas-oss2d8.ccs.ornl.gov kernel: [785564.401865] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST00f7_UUID lock: ffff880a522f1d80/0xf564dec7cde4ee02 lrc: 3/0,0 mode: PW/PW res: [0x5c7e19:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96faab expref: 770 pid: 118298 timeout: 5079935472 lvb_type: 0 Jul 17 20:28:26 atlas-oss2d2.ccs.ornl.gov kernel: [785564.494563] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0181_UUID lock: ffff88097901f240/0x6799745ceccc2782 lrc: 3/0,0 mode: PW/PW res: [0x52b781:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f548 expref: 630 pid: 117736 timeout: 5079936472 lvb_type: 0 Jul 17 20:28:26 atlas-oss2d2.ccs.ornl.gov kernel: [785564.627368] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST00f1_UUID lock: ffff880d76df8d80/0x6799745ceccc2ae6 lrc: 3/0,0 mode: PW/PW res: [0x5dc2dc:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f60c expref: 764 pid: 117736 timeout: 5079936473 lvb_type: 0 Jul 17 20:28:26 atlas-oss2g7.ccs.ornl.gov kernel: [785558.080205] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST022e_UUID lock: ffff880f14398480/0x6a005d9f55e7c071 lrc: 3/0,0 mode: PW/PW res: [0x5223fb:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f119 expref: 629 pid: 118124 timeout: 5079935469 lvb_type: 0 Jul 17 20:28:26 atlas-oss2g7.ccs.ornl.gov kernel: [785558.213186] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST02be_UUID lock: ffff880f4fff3900/0x6a005d9f55e7f6bf lrc: 3/0,0 mode: PW/PW res: [0x533d71:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fcf0 expref: 678 pid: 118103 timeout: 5079935472 lvb_type: 0 Jul 17 20:28:26 atlas-oss2g3.ccs.ornl.gov kernel: [785557.122251] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST022a_UUID lock: ffff880ab5c2ad80/0x697939cb1a6d5c3d lrc: 3/0,0 mode: PW/PW res: [0x56c927:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f985 expref: 656 pid: 117879 timeout: 5079933444 lvb_type: 0 Jul 17 20:28:26 atlas-oss1b3.ccs.ornl.gov kernel: [792822.374346] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST012a_UUID lock: ffff880d3116db40/0x166e16d10348172d lrc: 3/0,0 mode: PW/PW res: [0x582500:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f430 expref: 677 pid: 94625 timeout: 5087186379 lvb_type: 0 Jul 17 20:28:26 atlas-oss2c1.ccs.ornl.gov kernel: [785560.610213] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST00e8_UUID lock: ffff880e69f84b40/0x369867c371cfcaf0 lrc: 3/0,0 mode: PW/PW res: [0x57c374:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f2a8 expref: 697 pid: 25638 timeout: 5079936318 lvb_type: 0 Jul 17 20:28:26 atlas-oss2c1.ccs.ornl.gov kernel: [785560.735328] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0058_UUID lock: ffff8808fccee6c0/0x369867c371cfd4b3 lrc: 3/0,0 mode: PW/PW res: [0x55ce9d:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f4fb expref: 668 pid: 122852 timeout: 5079936319 lvb_type: 0 Jul 17 20:28:26 atlas-oss1h5.ccs.ornl.gov kernel: [792820.196202] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST00cc_UUID lock: ffff880c0b27e6c0/0x3293c92d9366b9b7 lrc: 3/0,0 mode: PW/PW res: [0x585e21:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fab2 expref: 630 pid: 39075 timeout: 5087184313 lvb_type: 0 Jul 17 20:28:27 atlas-oss2b6.ccs.ornl.gov kernel: [785563.300367] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0325_UUID lock: ffff88094f9cc900/0x17dfe8c68f94eda0 lrc: 3/0,0 mode: PW/PW res: [0x5379e1:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f2af expref: 671 pid: 118197 timeout: 5079936289 lvb_type: 0 Jul 17 20:28:27 atlas-oss2b6.ccs.ornl.gov kernel: [785563.431696] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0295_UUID lock: ffff880d915836c0/0x17dfe8c68f9506b3 lrc: 3/0,0 mode: PW/PW res: [0x5cac39:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f747 expref: 769 pid: 118024 timeout: 5079936291 lvb_type: 0 Jul 17 20:28:27 atlas-oss1g7.ccs.ornl.gov kernel: [712708.984383] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0156_UUID lock: ffff880d6048f6c0/0x880fb5b08d4025fe lrc: 3/0,0 mode: PW/PW res: [0x54d1c9:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f866 expref: 668 pid: 16757 timeout: 5007106276 lvb_type: 0 Jul 17 20:28:27 atlas-oss2h7.ccs.ornl.gov kernel: [785560.822294] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0116_UUID lock: ffff880b046ce480/0xc2ab54703efc9890 lrc: 3/0,0 mode: PW/PW res: [0x588c63:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f8c1 expref: 696 pid: 118179 timeout: 5079936270 lvb_type: 0 Jul 17 20:28:27 atlas-oss2b3.ccs.ornl.gov kernel: [785563.149359] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0292_UUID lock: ffff880969769b40/0xdf01f868f1ca3e17 lrc: 3/0,0 mode: PW/PW res: [0x55dc18:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f05c expref: 706 pid: 118416 timeout: 5079938264 lvb_type: 0 Jul 17 20:28:27 atlas-oss2i6.ccs.ornl.gov kernel: [785560.154683] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST023d_UUID lock: ffff880d66fe9900/0xb014c6aa0bbd4ac6 lrc: 3/0,0 mode: PW/PW res: [0x518ab0:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f86d expref: 638 pid: 23932 timeout: 5079936248 lvb_type: 0 Jul 17 20:28:27 atlas-oss1h8.ccs.ornl.gov kernel: [792817.884533] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST039f_UUID lock: ffff880ce2ecab40/0x775e4efc0215e055 lrc: 3/0,0 mode: PW/PW res: [0x58cb5f:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f804 expref: 702 pid: 92604 timeout: 5087184235 lvb_type: 0 Jul 17 20:28:27 atlas-oss1f5.ccs.ornl.gov kernel: [792813.669955] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST002c_UUID lock: ffff880a12a38900/0xacfaa93b501c8f78 lrc: 3/0,0 mode: PW/PW res: [0x54cc32:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f9e7 expref: 716 pid: 38779 timeout: 5087184214 lvb_type: 0 Jul 17 20:28:27 atlas-oss2c3.ccs.ornl.gov kernel: [785566.328388] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST03ba_UUID lock: ffff880daf511b40/0x950cd86563b23d5e lrc: 3/0,0 mode: PW/PW res: [0x5673b2:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f61a expref: 699 pid: 122314 timeout: 5079938186 lvb_type: 0 Jul 17 20:28:27 atlas-oss1c7.ccs.ornl.gov kernel: [792818.028171] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST01c6_UUID lock: ffff880ebb2ab900/0x33255120d4246978 lrc: 3/0,0 mode: PW/PW res: [0x54e753:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f1e4 expref: 659 pid: 38685 timeout: 5087184314 lvb_type: 0 Jul 17 20:28:27 atlas-oss1c7.ccs.ornl.gov kernel: [792818.156413] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST02e6_UUID lock: ffff880bc6484480/0x33255120d42470c5 lrc: 3/0,0 mode: PW/PW res: [0x4ff91c:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f4ed expref: 623 pid: 38664 timeout: 5087184315 lvb_type: 0 Jul 17 20:28:27 atlas-oss1e2.ccs.ornl.gov kernel: [792820.981138] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0141_UUID lock: ffff880b35072240/0x834905efc602a134 lrc: 3/0,0 mode: PW/PW res: [0x4f67a2:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f36c expref: 626 pid: 39810 timeout: 5087186184 lvb_type: 0 Jul 17 20:28:27 atlas-oss1e1.ccs.ornl.gov kernel: [792822.091045] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST01d0_UUID lock: ffff880e84ac0240/0xf4707858d5fdd201 lrc: 3/0,0 mode: PW/PW res: [0x537997:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f2fc expref: 650 pid: 93844 timeout: 5087186172 lvb_type: 0 Jul 17 20:28:27 atlas-oss1e1.ccs.ornl.gov kernel: [792822.217876] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST01d0_UUID lock: ffff880d2bec8b40/0xf4707858d5fde7cc lrc: 3/0,0 mode: PW/PW res: [0x5379bc:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f740 expref: 650 pid: 37838 timeout: 5087186173 lvb_type: 0 Jul 17 20:28:27 atlas-oss1h4.ccs.ornl.gov kernel: [792817.701383] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST01eb_UUID lock: ffff880c68d65480/0xc6629348e0358353 lrc: 3/0,0 mode: PW/PW res: [0x525094:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f23f expref: 665 pid: 38778 timeout: 5087184151 lvb_type: 0 Jul 17 20:28:27 atlas-oss1h4.ccs.ornl.gov kernel: [792817.826296] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST003b_UUID lock: ffff880b8bce7b40/0xc6629348e0358903 lrc: 3/0,0 mode: PW/PW res: [0x59ba25:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f48b expref: 719 pid: 94128 timeout: 5087184152 lvb_type: 0 Jul 17 20:28:27 atlas-oss2e3.ccs.ornl.gov kernel: [784933.078577] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST02aa_UUID lock: ffff880cffc2ed80/0xbc1626a312f27d00 lrc: 3/0,0 mode: PW/PW res: [0x554e3c:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f732 expref: 677 pid: 20302 timeout: 5079307131 lvb_type: 0 Jul 17 20:28:27 atlas-oss2g4.ccs.ornl.gov kernel: [785561.950875] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST007b_UUID lock: ffff880cc9fb1240/0x530d610ccfd58367 lrc: 3/0,0 mode: PW/PW res: [0x5b3b62:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f055 expref: 730 pid: 117658 timeout: 5079938116 lvb_type: 0 Jul 17 20:28:27 atlas-oss2g4.ccs.ornl.gov kernel: [785562.080019] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST010b_UUID lock: ffff880a323ee900/0x530d610ccfd5b8b2 lrc: 3/0,0 mode: PW/PW res: [0x58c91e:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fbd8 expref: 697 pid: 117818 timeout: 5079938120 lvb_type: 0 Jul 17 20:28:27 atlas-oss1c4.ccs.ornl.gov kernel: [792820.335319] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0373_UUID lock: ffff8809c4d05900/0x8ac2775f11ecb19f lrc: 3/0,0 mode: PW/PW res: [0x5223d2:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f79b expref: 657 pid: 38003 timeout: 5087186091 lvb_type: 0 Jul 17 20:28:27 atlas-oss2h5.ccs.ornl.gov kernel: [785556.028594] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST03e4_UUID lock: ffff8809089f5000/0xc0c9ab91e46bb0bc lrc: 3/0,0 mode: PW/PW res: [0x5b7661:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f4e6 expref: 717 pid: 113808 timeout: 5079935076 lvb_type: 0 Jul 17 20:28:27 atlas-oss2h5.ccs.ornl.gov kernel: [785556.154955] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0234_UUID lock: ffff880936805000/0xc0c9ab91e46bc481 lrc: 3/0,0 mode: PW/PW res: [0x550c29:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f9ee expref: 627 pid: 113692 timeout: 5079935077 lvb_type: 0 Jul 17 20:28:27 atlas-oss2h6.ccs.ornl.gov kernel: [785510.232623] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0355_UUID lock: ffff880a83c15000/0xf2683ada942494b3 lrc: 3/0,0 mode: PW/PW res: [0x563aaf:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f246 expref: 679 pid: 113499 timeout: 5079884074 lvb_type: 0 Jul 17 20:28:27 atlas-oss1f8.ccs.ornl.gov kernel: [792820.027974] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST01df_UUID lock: ffff880b0fe0cb40/0x752cebe8c56edd07 lrc: 3/0,0 mode: PW/PW res: [0x5764e3:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f97e expref: 721 pid: 38849 timeout: 5087184033 lvb_type: 0 Jul 17 20:28:27 atlas-oss1e3.ccs.ornl.gov kernel: [792816.665730] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0262_UUID lock: ffff8809483a76c0/0xee73bbd7c9f78352 lrc: 3/0,0 mode: PW/PW res: [0x5348cd:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f0b7 expref: 602 pid: 38738 timeout: 5087184026 lvb_type: 0 Jul 17 20:28:27 atlas-oss2f5.ccs.ornl.gov kernel: [785568.144805] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0074_UUID lock: ffff8809645acd80/0x73b9bd9f0c878ccc lrc: 3/0,0 mode: PW/PW res: [0x4c0c15:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f613 expref: 587 pid: 26087 timeout: 5079938025 lvb_type: 0 Jul 17 20:28:27 atlas-oss2f5.ccs.ornl.gov kernel: [785568.274520] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0224_UUID lock: ffff8808f0119d80/0x73b9bd9f0c879102 lrc: 3/0,0 mode: PW/PW res: [0x55ddb5:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f6d0 expref: 686 pid: 118144 timeout: 5079938025 lvb_type: 0 Jul 17 20:28:27 atlas-oss2b5.ccs.ornl.gov kernel: [785559.483039] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0054_UUID lock: ffff880ef921d000/0xc0a92f83399edb0c lrc: 3/0,0 mode: PW/PW res: [0x55343f:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f365 expref: 670 pid: 117990 timeout: 5079935024 lvb_type: 0 Jul 17 20:28:27 atlas-oss2c5.ccs.ornl.gov kernel: [785565.726084] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST029c_UUID lock: ffff880c7555d000/0xaec6cb2f63758ad9 lrc: 3/0,0 mode: PW/PW res: [0x520d31:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f80b expref: 592 pid: 122761 timeout: 5079935014 lvb_type: 0 Jul 17 20:28:27 atlas-oss1h3.ccs.ornl.gov kernel: [792814.937379] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST039a_UUID lock: ffff880ee9eaa6c0/0xe2b024449fca88b9 lrc: 3/0,0 mode: PW/PW res: [0x518cfa:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f112 expref: 613 pid: 38549 timeout: 5087183004 lvb_type: 0 Jul 17 20:28:27 atlas-oss2f8.ccs.ornl.gov kernel: [785563.174661] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0197_UUID lock: ffff880ab695fd80/0x415a8c174fbe6887 lrc: 3/0,0 mode: PW/PW res: [0x562f70:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96fb14 expref: 673 pid: 118055 timeout: 5079935005 lvb_type: 0 Jul 17 20:28:27 atlas-oss1h2.ccs.ornl.gov kernel: [792817.145159] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0309_UUID lock: ffff880da6cb6d80/0x24d2a2a170ee73dc lrc: 3/0,0 mode: PW/PW res: [0x56348f:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f17b expref: 672 pid: 38875 timeout: 5087184004 lvb_type: 0 Jul 17 20:28:27 atlas-oss1i3.ccs.ornl.gov kernel: [792814.221332] LustreError: 57:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0312_UUID lock: ffff880a9511b900/0x478cea066e1ab0c6 lrc: 3/0,0 mode: PW/PW res: [0x545d2c:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f6e5 expref: 734 pid: 38381 timeout: 5087184004 lvb_type: 0 Jul 17 20:28:27 atlas-oss1h7.ccs.ornl.gov kernel: [792818.773578] LustreError: 53:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST003e_UUID lock: ffff8808ef5a9900/0x286490182ffc7b2e lrc: 3/0,0 mode: PW/PW res: [0x592abd:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f7f6 expref: 681 pid: 38426 timeout: 5087186000 lvb_type: 0 Jul 17 20:28:27 atlas-oss1h7.ccs.ornl.gov kernel: [792818.906941] LustreError: 53:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST003e_UUID lock: ffff88091d790000/0x286490182ffc8672 lrc: 3/0,0 mode: PW/PW res: [0x592acc:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->1048575) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac96f98c expref: 681 pid: 38272 timeout: 5087186000 lvb_type: 0 Jul 17 20:29:14 atlas-oss1f1.ccs.ornl.gov kernel: [792865.707522] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 92s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST02f8_UUID lock: ffff880c88279b40/0x11123c2f0d4a482f lrc: 3/0,0 mode: PW/PW res: [0x562506:0x0:0x0].0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x20 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac8c7aa4 expref: 643 pid: 38697 timeout: 5087232492 lvb_type: 0 Jul 17 20:30:42 atlas-mds3.ccs.ornl.gov kernel: [10018.594246] Lustre: atlas2-MDT0000: Denying connection for new client 9f60235a-8b2c-2e12-f871-36864c4e6362 (at 1105@gni100), waiting for all 20156 known clients (19151 recovered, 981 in progress, and 24 evicted) to recover in 1:56 Jul 17 20:30:42 atlas-mds3.ccs.ornl.gov kernel: [10018.617435] Lustre: Skipped 27 previous similar messages Jul 17 20:31:25 atlas-oss1a2.ccs.ornl.gov kernel: [792993.612218] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0091_UUID lock: ffff880f4c751240/0x4f5405432a3ec4da lrc: 3/0,0 mode: PR/PR res: [0x5811b8:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac883228 expref: 698 pid: 95153 timeout: 5087362288 lvb_type: 1 Jul 17 20:33:21 atlas-mds3.ccs.ornl.gov kernel: [10178.548880] Lustre: atlas2-MDT0000: Client 25eca314-2213-dba4-4e7c-ae1acb406793 (at 15@gni2) reconnecting, waiting for 20156 clients in recovery for 0:-42 Jul 17 20:33:21 atlas-mds3.ccs.ornl.gov kernel: [10178.566474] Lustre: Skipped 372 previous similar messages Jul 17 20:33:21 atlas-mds3.ccs.ornl.gov kernel: [10178.572627] Lustre: atlas2-MDT0000: Client 25eca314-2213-dba4-4e7c-ae1acb406793 (at 15@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 20:33:21 atlas-mds3.ccs.ornl.gov kernel: [10178.587340] Lustre: Skipped 372 previous similar messages Jul 17 20:33:38 atlas-mds3.ccs.ornl.gov kernel: [10194.664220] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 3791@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff883fcc151900/0xf35a0587ba9997f6 lrc: 3/0,0 mode: PR/PR res: [0x2000d1f90:0x3123:0x0].0 bits 0x13 rrc: 156 type: IBT flags: 0x20 nid: 3791@gni100 remote: 0xb462a16f4e764390 expref: 130 pid: 16192 timeout: 4304850028 lvb_type: 0 used 0 Jul 17 20:33:38 atlas-mds3.ccs.ornl.gov kernel: [10194.706161] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 1 previous similar message Jul 17 20:33:38 atlas-mds3.ccs.ornl.gov kernel: [10194.724797] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 20:33:38 atlas-mds3.ccs.ornl.gov kernel: [10194.739355] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff881d05c0a480/0xf35a0587ba9a0890 lrc: 3/0,0 mode: PR/PR res: [0x2000d375b:0x1b3e:0x0].0 bits 0x13 rrc: 27 type: IBT flags: 0x20 nid: 12977@gni100 remote: 0x488746107ff0deaa expref: 129 pid: 15654 timeout: 0 lvb_type: 0 used 0 Jul 17 20:33:38 atlas-mds3.ccs.ornl.gov kernel: [10194.777771] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) Skipped 3 previous similar messages Jul 17 20:39:16 atlas-mgs1.ccs.ornl.gov kernel: [808727.435007] Lustre: MGS: Client 5c4122e7-aa02-5273-d8bf-c0e12d67d8f8 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-linkfarm.ccs.ornl.gov kernel: [300828.098723] Lustre: linkfarm-MDT0000: Client e0b7d7d4-53f5-ebcc-736c-47a1b154601e (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2i5.ccs.ornl.gov kernel: [786212.174659] Lustre: atlas1-OST01ac: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2e8.ccs.ornl.gov kernel: [786217.695731] Lustre: atlas1-OST021f: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2f5.ccs.ornl.gov kernel: [786217.567973] Lustre: atlas1-OST0194: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1c3.ccs.ornl.gov kernel: [793467.235328] Lustre: atlas1-OST01c2: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1c6.ccs.ornl.gov kernel: [793461.792010] Lustre: atlas1-OST01c5: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2c2.ccs.ornl.gov kernel: [786211.852048] Lustre: atlas1-OST0209: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1g8.ccs.ornl.gov kernel: [713355.389772] Lustre: atlas1-OST01e7: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1b4.ccs.ornl.gov kernel: [793467.855129] Lustre: atlas1-OST01bb: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2c8.ccs.ornl.gov kernel: [786217.218561] Lustre: atlas1-OST03bf: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2b5.ccs.ornl.gov kernel: [786208.902066] Lustre: atlas1-OST0294: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1h2.ccs.ornl.gov kernel: [793466.550332] Lustre: atlas1-OST0399: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2h8.ccs.ornl.gov kernel: [785500.865023] Lustre: atlas1-OST02c7: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2c1.ccs.ornl.gov kernel: [786210.324057] Lustre: atlas1-OST0178: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1d8.ccs.ornl.gov kernel: [793466.444256] Lustre: atlas1-OST037f: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2f7.ccs.ornl.gov kernel: [786216.672086] Lustre: atlas1-OST03d6: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2a7.ccs.ornl.gov kernel: [786213.943596] Lustre: atlas1-OST03ae: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1f3.ccs.ornl.gov kernel: [793467.173599] Lustre: atlas1-OST038a: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1i6.ccs.ornl.gov kernel: [793466.487180] Lustre: atlas1-OST0285: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2d1.ccs.ornl.gov kernel: [786211.332036] Lustre: atlas1-OST0180: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3a3.ccs.ornl.gov kernel: [1928713.574246] Lustre: atlas2-OST0092: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3i3.ccs.ornl.gov kernel: [1928711.824675] Lustre: atlas2-OST0042: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3f2.ccs.ornl.gov kernel: [1928711.472069] Lustre: atlas2-OST00b9: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1a2.ccs.ornl.gov kernel: [793464.924482] Lustre: atlas1-OST02d1: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4h4.ccs.ornl.gov kernel: [1928735.890730] Lustre: atlas2-OST0113: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3i3.ccs.ornl.gov kernel: [1928711.848188] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss4i3.ccs.ornl.gov kernel: [1928708.543917] Lustre: atlas2-OST008a: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3g8.ccs.ornl.gov kernel: [1927314.429421] Lustre: atlas2-OST0157: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4d3.ccs.ornl.gov kernel: [1928712.817759] Lustre: atlas2-OST00f2: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4b8.ccs.ornl.gov kernel: [1928718.239919] Lustre: atlas2-OST0177: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3g6.ccs.ornl.gov kernel: [1927324.834294] Lustre: atlas2-OST00c5: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3b2.ccs.ornl.gov kernel: [1928716.370977] Lustre: atlas2-OST0099: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4i3.ccs.ornl.gov kernel: [1928708.567407] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss1f6.ccs.ornl.gov kernel: [793413.127320] Lustre: atlas1-OST00bd: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3g4.ccs.ornl.gov kernel: [1927326.982834] Lustre: atlas2-OST0033: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4e1.ccs.ornl.gov kernel: [1928714.194621] Lustre: atlas2-OST0068: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1b6.ccs.ornl.gov kernel: [793464.988036] Lustre: atlas1-OST01bd: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1h3.ccs.ornl.gov kernel: [793464.341197] Lustre: atlas1-OST01ea: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2d5.ccs.ornl.gov kernel: [786213.136104] Lustre: atlas1-OST0214: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2f8.ccs.ornl.gov kernel: [786212.575988] Lustre: atlas1-OST03d7: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4g5.ccs.ornl.gov kernel: [1928716.453334] Lustre: atlas2-OST02bc: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3i4.ccs.ornl.gov kernel: [1928722.266537] Lustre: atlas2-OST01f3: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4e1.ccs.ornl.gov kernel: [1928714.218671] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss3g4.ccs.ornl.gov kernel: [1927327.006990] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss4d4.ccs.ornl.gov kernel: [1928736.841062] Lustre: atlas2-OST0213: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2f8.ccs.ornl.gov kernel: [786212.600272] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss2i8.ccs.ornl.gov kernel: [786215.265577] Lustre: atlas1-OST02cf: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2c5.ccs.ornl.gov kernel: [786215.139292] Lustre: atlas1-OST020c: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4a1.ccs.ornl.gov kernel: [1928714.302819] Lustre: atlas2-OST01f8: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4b7.ccs.ornl.gov kernel: [1928710.418118] Lustre: atlas2-OST0176: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3a2.ccs.ornl.gov kernel: [1928715.901072] Lustre: atlas2-OST02d1: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1d4.ccs.ornl.gov kernel: [793464.384353] Lustre: atlas1-OST01cb: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4a6.ccs.ornl.gov kernel: [1928718.458366] Lustre: atlas2-OST031d: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3b3.ccs.ornl.gov kernel: [1928709.291767] Lustre: atlas2-OST012a: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1e6.ccs.ornl.gov kernel: [793464.373222] Lustre: atlas1-OST01d5: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2g5.ccs.ornl.gov kernel: [786212.869254] Lustre: atlas1-OST022c: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2b3.ccs.ornl.gov kernel: [786212.809776] Lustre: atlas1-OST00e2: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3g2.ccs.ornl.gov kernel: [1927313.014760] Lustre: atlas2-OST01e1: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4d8.ccs.ornl.gov kernel: [1928729.512753] Lustre: atlas2-OST0217: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3h6.ccs.ornl.gov kernel: [1928716.756918] Lustre: atlas2-OST030d: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3b3.ccs.ornl.gov kernel: [1928709.316126] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss4c2.ccs.ornl.gov kernel: [1928720.307019] Lustre: atlas2-OST0329: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3c2.ccs.ornl.gov kernel: [1928718.434850] Lustre: atlas2-OST01c1: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3b5.ccs.ornl.gov kernel: [1928729.460683] Lustre: atlas2-OST000c: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2f3.ccs.ornl.gov kernel: [786209.977640] Lustre: atlas1-OST02b2: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3d2.ccs.ornl.gov kernel: [1928718.072634] Lustre: atlas2-OST0019: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3e4.ccs.ornl.gov kernel: [1928730.720651] Lustre: atlas2-OST0143: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3b5.ccs.ornl.gov kernel: [1928729.485978] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss3d2.ccs.ornl.gov kernel: [1928718.097908] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss4f5.ccs.ornl.gov kernel: [1928714.309088] Lustre: atlas2-OST0224: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1f1.ccs.ornl.gov kernel: [793467.562138] Lustre: atlas1-OST0028: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2c4.ccs.ornl.gov kernel: [786209.413347] Lustre: atlas1-OST00eb: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4c5.ccs.ornl.gov kernel: [1928712.083490] Lustre: atlas2-OST005c: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2f4.ccs.ornl.gov kernel: [786216.071748] Lustre: atlas1-OST0223: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4a3.ccs.ornl.gov kernel: [1928732.442620] Lustre: atlas2-OST031a: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1e5.ccs.ornl.gov kernel: [793462.752578] Lustre: atlas1-OST0384: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4f6.ccs.ornl.gov kernel: [1928711.132511] Lustre: atlas2-OST0345: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4f7.ccs.ornl.gov kernel: [1928715.429327] Lustre: atlas2-OST0196: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4h8.ccs.ornl.gov kernel: [1928707.277993] Lustre: atlas2-OST0087: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1b2.ccs.ornl.gov kernel: [793466.332611] Lustre: atlas1-OST0099: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1e7.ccs.ornl.gov kernel: [793467.266786] Lustre: atlas1-OST00b6: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1c1.ccs.ornl.gov kernel: [793468.247785] Lustre: atlas1-OST0250: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3i5.ccs.ornl.gov kernel: [715024.225831] Lustre: atlas2-OST01f4: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1c8.ccs.ornl.gov kernel: [793466.814311] Lustre: atlas1-OST01c7: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3d1.ccs.ornl.gov kernel: [1928708.997800] Lustre: atlas2-OST0138: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4c5.ccs.ornl.gov kernel: [1928712.109074] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss1d6.ccs.ornl.gov kernel: [793469.860591] Lustre: atlas1-OST00ad: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1c4.ccs.ornl.gov kernel: [793469.829015] Lustre: atlas1-OST00a3: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3b7.ccs.ornl.gov kernel: [1928721.277781] Lustre: atlas2-OST009e: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4h8.ccs.ornl.gov kernel: [1928707.303561] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss3f1.ccs.ornl.gov kernel: [1928720.236980] Lustre: atlas2-OST02f8: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3d4.ccs.ornl.gov kernel: [715024.543967] Lustre: atlas2-OST01cb: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3d3.ccs.ornl.gov kernel: [1928723.245644] Lustre: atlas2-OST02ea: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3d6.ccs.ornl.gov kernel: [1928730.853202] Lustre: atlas2-OST025d: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2i2.ccs.ornl.gov kernel: [786210.624279] Lustre: atlas1-OST03e9: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3c4.ccs.ornl.gov kernel: [1928711.632415] Lustre: atlas2-OST01c3: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4c6.ccs.ornl.gov kernel: [1928717.292393] Lustre: atlas2-OST005d: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1d3.ccs.ornl.gov kernel: [793468.725883] Lustre: atlas1-OST013a: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2i3.ccs.ornl.gov kernel: [786216.241323] Lustre: atlas1-OST011a: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3g1.ccs.ornl.gov kernel: [1927306.291073] Lustre: atlas2-OST0270: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3h4.ccs.ornl.gov kernel: [1928720.175181] Lustre: atlas2-OST01eb: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4f8.ccs.ornl.gov kernel: [1928715.621173] Lustre: atlas2-OST0107: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2i4.ccs.ornl.gov kernel: [786211.078798] Lustre: atlas1-OST011b: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2a6.ccs.ornl.gov kernel: [786215.073820] Lustre: atlas1-OST01fd: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3f4.ccs.ornl.gov kernel: [1928718.537839] Lustre: atlas2-OST02fb: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4g2.ccs.ornl.gov kernel: [1928727.208990] Lustre: atlas2-OST0079: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4a5.ccs.ornl.gov kernel: [1928735.485955] Lustre: atlas2-OST01fc: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4a2.ccs.ornl.gov kernel: [1928731.089603] Lustre: atlas2-OST0319: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4g7.ccs.ornl.gov kernel: [1928738.798277] Lustre: atlas2-OST010e: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4i1.ccs.ornl.gov kernel: [1928716.481202] Lustre: atlas2-OST01a8: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4i8.ccs.ornl.gov kernel: [1928730.131506] Lustre: atlas2-OST02cf: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3g3.ccs.ornl.gov kernel: [1927313.785411] Lustre: atlas2-OST0152: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4i7.ccs.ornl.gov kernel: [1928729.745470] Lustre: atlas2-OST011e: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3i7.ccs.ornl.gov kernel: [1928713.511848] Lustre: atlas2-OST00d6: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3d7.ccs.ornl.gov kernel: [1928717.245589] Lustre: atlas2-OST013e: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2a8.ccs.ornl.gov kernel: [786212.777466] Lustre: atlas1-OST01ff: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1f2.ccs.ornl.gov kernel: [793468.043488] Lustre: atlas1-OST0269: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1i3.ccs.ornl.gov kernel: [793463.622554] Lustre: atlas1-OST03a2: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4c7.ccs.ornl.gov kernel: [1928716.245626] Lustre: atlas2-OST032e: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3a6.ccs.ornl.gov kernel: [1928719.013926] Lustre: atlas2-OST01b5: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1f7.ccs.ornl.gov kernel: [792638.155619] Lustre: atlas1-OST01de: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2g8.ccs.ornl.gov kernel: [786218.773815] Lustre: atlas1-OST007f: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2g4.ccs.ornl.gov kernel: [786211.463022] Lustre: atlas1-OST019b: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4e8.ccs.ornl.gov kernel: [1928729.390497] Lustre: atlas2-OST018f: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3h3.ccs.ornl.gov kernel: [1928711.939029] Lustre: atlas2-OST027a: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1c5.ccs.ornl.gov kernel: [793462.309642] Lustre: atlas1-OST0374: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4a7.ccs.ornl.gov kernel: [1928728.111107] Lustre: atlas2-OST028e: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2b7.ccs.ornl.gov kernel: [786215.440627] Lustre: atlas1-OST00e6: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4e7.ccs.ornl.gov kernel: [1928713.735808] Lustre: atlas2-OST021e: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4a8.ccs.ornl.gov kernel: [1928722.454617] Lustre: atlas2-OST028f: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1b5.ccs.ornl.gov kernel: [793470.520078] Lustre: atlas1-OST009c: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4i2.ccs.ornl.gov kernel: [1928739.386300] Lustre: atlas2-OST0239: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2b2.ccs.ornl.gov kernel: [786208.038701] Lustre: atlas1-OST0171: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2f6.ccs.ornl.gov kernel: [786219.074664] Lustre: atlas1-OST02b5: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4b6.ccs.ornl.gov kernel: [1928713.943558] Lustre: atlas2-OST0055: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4b6.ccs.ornl.gov kernel: [1928713.971409] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss2e7.ccs.ornl.gov kernel: [786212.905697] Lustre: atlas1-OST021e: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3g5.ccs.ornl.gov kernel: [1927317.987055] Lustre: atlas2-OST01e4: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4d2.ccs.ornl.gov kernel: [1928721.484551] Lustre: atlas2-OST0331: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4i4.ccs.ornl.gov kernel: [1928721.318791] Lustre: atlas2-OST023b: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3e1.ccs.ornl.gov kernel: [1928719.139500] Lustre: atlas2-OST00b0: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3h2.ccs.ornl.gov kernel: [1928719.429253] Lustre: atlas2-OST0159: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3h1.ccs.ornl.gov kernel: [1928704.999828] Lustre: atlas2-OST01e8: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4g3.ccs.ornl.gov kernel: [1928727.569638] Lustre: atlas2-OST010a: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4b5.ccs.ornl.gov kernel: [1928734.056231] Lustre: atlas2-OST0174: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1a8.ccs.ornl.gov kernel: [793465.301981] Lustre: atlas1-OST0247: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4d1.ccs.ornl.gov kernel: [1928712.380516] Lustre: atlas2-OST0210: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4i5.ccs.ornl.gov kernel: [1928724.370364] Lustre: atlas2-OST035c: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3e7.ccs.ornl.gov kernel: [1928718.771603] Lustre: atlas2-OST01d6: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4b4.ccs.ornl.gov kernel: [1928714.446125] Lustre: atlas2-OST0203: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1e3.ccs.ornl.gov kernel: [793466.092896] Lustre: atlas1-OST01d2: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3b4.ccs.ornl.gov kernel: [1928728.016646] Lustre: atlas2-OST01bb: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4c3.ccs.ornl.gov kernel: [1928713.124121] Lustre: atlas2-OST005a: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4c3.ccs.ornl.gov kernel: [1928713.152559] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss3a7.ccs.ornl.gov kernel: [1928714.484055] Lustre: atlas2-OST0246: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4g6.ccs.ornl.gov kernel: [1928726.134834] Lustre: atlas2-OST034d: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3c3.ccs.ornl.gov kernel: [1928709.027340] Lustre: atlas2-OST02e2: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4h2.ccs.ornl.gov kernel: [1928737.731849] Lustre: atlas2-OST0111: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2h1.ccs.ornl.gov kernel: [786215.325881] Lustre: atlas1-OST01a0: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2a5.ccs.ornl.gov kernel: [786215.059380] Lustre: atlas1-OST016c: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1d1.ccs.ornl.gov kernel: [793471.782020] Lustre: atlas1-OST01c8: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3c5.ccs.ornl.gov kernel: [1928723.029785] Lustre: atlas2-OST0134: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2h5.ccs.ornl.gov kernel: [786205.496153] Lustre: atlas1-OST01a4: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1i1.ccs.ornl.gov kernel: [793467.561937] Lustre: atlas1-OST0160: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1a4.ccs.ornl.gov kernel: [793456.621782] Lustre: atlas1-OST0123: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2h6.ccs.ornl.gov kernel: [786159.702319] Lustre: atlas1-OST0235: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3a5.ccs.ornl.gov kernel: [1928709.878695] Lustre: atlas2-OST0094: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3h8.ccs.ornl.gov kernel: [1928713.218801] Lustre: atlas2-OST003f: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3h8.ccs.ornl.gov kernel: [1928713.247586] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss3h5.ccs.ornl.gov kernel: [1928736.210122] Lustre: atlas2-OST003c: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3h5.ccs.ornl.gov kernel: [1928736.239062] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss4e4.ccs.ornl.gov kernel: [1928725.120486] Lustre: atlas2-OST033b: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1a1.ccs.ornl.gov kernel: [793464.651161] Lustre: atlas1-OST0240: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1d5.ccs.ornl.gov kernel: [793468.066641] Lustre: atlas1-OST001c: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4e5.ccs.ornl.gov kernel: [1928717.482244] Lustre: atlas2-OST00fc: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2g2.ccs.ornl.gov kernel: [786211.667513] Lustre: atlas1-OST0229: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4f1.ccs.ornl.gov kernel: [1928720.571710] Lustre: atlas2-OST02b0: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2b6.ccs.ornl.gov kernel: [786212.987389] Lustre: atlas1-OST0205: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3b8.ccs.ornl.gov kernel: [1928730.620074] Lustre: atlas2-OST009f: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3i1.ccs.ornl.gov kernel: [1928716.531128] Lustre: atlas2-OST01f0: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4f4.ccs.ornl.gov kernel: [1928716.828161] Lustre: atlas2-OST0193: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4h5.ccs.ornl.gov kernel: [1928718.914504] Lustre: atlas2-OST0084: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4h5.ccs.ornl.gov kernel: [1928718.943852] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss4c1.ccs.ornl.gov kernel: [1928721.910527] Lustre: atlas2-OST0298: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3b1.ccs.ornl.gov kernel: [1928723.080081] Lustre: atlas2-OST01b8: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1h4.ccs.ornl.gov kernel: [793467.253823] Lustre: atlas1-OST039b: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4f2.ccs.ornl.gov kernel: [1928719.959191] Lustre: atlas2-OST0341: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2i1.ccs.ornl.gov kernel: [785497.167003] Lustre: atlas1-OST02c8: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4e3.ccs.ornl.gov kernel: [1928722.206535] Lustre: atlas2-OST006a: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4d6.ccs.ornl.gov kernel: [1928716.464435] Lustre: atlas2-OST02a5: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3f6.ccs.ornl.gov kernel: [815420.986700] Lustre: atlas2-OST026d: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3f3.ccs.ornl.gov kernel: [1928729.969957] Lustre: atlas2-OST00ba: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1g1.ccs.ornl.gov kernel: [713355.277883] Lustre: atlas1-OST0030: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1f4.ccs.ornl.gov kernel: [793467.474195] Lustre: atlas1-OST026b: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3f5.ccs.ornl.gov kernel: [815408.211386] Lustre: atlas2-OST014c: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2d6.ccs.ornl.gov kernel: [786211.833466] Lustre: atlas1-OST0185: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4e3.ccs.ornl.gov kernel: [1928722.236403] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss1b1.ccs.ornl.gov kernel: [793465.568079] Lustre: atlas1-OST0248: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1e4.ccs.ornl.gov kernel: [793467.377719] Lustre: atlas1-OST0383: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3f7.ccs.ornl.gov kernel: [815047.476671] Lustre: atlas2-OST002e: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3c6.ccs.ornl.gov kernel: [1928708.581024] Lustre: atlas2-OST02e5: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4h3.ccs.ornl.gov kernel: [1928729.128584] Lustre: atlas2-OST02c2: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1a5.ccs.ornl.gov kernel: [792711.087253] Lustre: atlas1-OST0124: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3i8.ccs.ornl.gov kernel: [1928730.358928] Lustre: atlas2-OST00d7: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3h7.ccs.ornl.gov kernel: [1928723.127647] Lustre: atlas2-OST01ee: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2a1.ccs.ornl.gov kernel: [786216.370057] Lustre: atlas1-OST00d8: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4c8.ccs.ornl.gov kernel: [1928708.202551] Lustre: atlas2-OST017f: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3a1.ccs.ornl.gov kernel: [1928711.157908] Lustre: atlas2-OST0120: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3e5.ccs.ornl.gov kernel: [1928717.606955] Lustre: atlas2-OST0144: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2b8.ccs.ornl.gov kernel: [786213.554812] Lustre: atlas1-OST0297: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2d4.ccs.ornl.gov kernel: [786160.746228] Lustre: atlas1-OST0063: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4h7.ccs.ornl.gov kernel: [1928724.682169] Lustre: atlas2-OST0086: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3d5.ccs.ornl.gov kernel: [1928719.883659] Lustre: atlas2-OST00ac: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2f2.ccs.ornl.gov kernel: [786212.088997] Lustre: atlas1-OST03d1: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3d8.ccs.ornl.gov kernel: [1928722.986534] Lustre: atlas2-OST013f: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1b3.ccs.ornl.gov kernel: [793472.158205] Lustre: atlas1-OST000a: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4b2.ccs.ornl.gov kernel: [1928739.309122] Lustre: atlas2-OST0171: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1g5.ccs.ornl.gov kernel: [713359.460045] Lustre: atlas1-OST0034: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2a4.ccs.ornl.gov kernel: [786216.493142] Lustre: atlas1-OST01fb: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1g6.ccs.ornl.gov kernel: [713357.149535] Lustre: atlas1-OST0155: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2f2.ccs.ornl.gov kernel: [786212.119820] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss4a4.ccs.ornl.gov kernel: [1928717.926268] Lustre: atlas2-OST004b: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4h7.ccs.ornl.gov kernel: [1928724.712676] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss1i5.ccs.ornl.gov kernel: [793462.295335] Lustre: atlas1-OST01f4: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3c8.ccs.ornl.gov kernel: [1928721.435744] Lustre: atlas2-OST00a7: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2h4.ccs.ornl.gov kernel: [786213.235141] Lustre: atlas1-OST01a3: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2b1.ccs.ornl.gov kernel: [786213.108981] Lustre: atlas1-OST0050: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4g1.ccs.ornl.gov kernel: [1928713.436968] Lustre: atlas2-OST0198: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1h8.ccs.ornl.gov kernel: [793467.518626] Lustre: atlas1-OST027f: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1a6.ccs.ornl.gov kernel: [793468.450022] Lustre: atlas1-OST0245: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4a4.ccs.ornl.gov kernel: [1928717.957125] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss1h6.ccs.ornl.gov kernel: [793468.436079] Lustre: atlas1-OST027d: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2i7.ccs.ornl.gov kernel: [786209.844409] Lustre: atlas1-OST02ce: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2h2.ccs.ornl.gov kernel: [786216.610817] Lustre: atlas1-OST0231: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4e2.ccs.ornl.gov kernel: [1928730.269994] Lustre: atlas2-OST0189: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1g7.ccs.ornl.gov kernel: [713358.660074] Lustre: atlas1-OST00c6: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1e2.ccs.ornl.gov kernel: [793470.569829] Lustre: atlas1-OST00b1: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2d3.ccs.ornl.gov kernel: [786208.414902] Lustre: atlas1-OST00f2: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2a3.ccs.ornl.gov kernel: [786218.759027] Lustre: atlas1-OST03aa: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4g8.ccs.ornl.gov kernel: [1928715.457737] Lustre: atlas2-OST019f: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2h7.ccs.ornl.gov kernel: [786210.486517] Lustre: atlas1-OST03e6: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1i7.ccs.ornl.gov kernel: [793466.515617] Lustre: atlas1-OST0286: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2a3.ccs.ornl.gov kernel: [786218.790462] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss3a4.ccs.ornl.gov kernel: [1928706.293938] Lustre: atlas2-OST0123: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2i6.ccs.ornl.gov kernel: [786209.796377] Lustre: atlas1-OST02cd: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4b3.ccs.ornl.gov kernel: [1928710.631075] Lustre: atlas2-OST0292: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3e2.ccs.ornl.gov kernel: [1928715.754651] Lustre: atlas2-OST01d1: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3e6.ccs.ornl.gov kernel: [1928713.920774] Lustre: atlas2-OST0025: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2h7.ccs.ornl.gov kernel: [786210.518052] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss4d5.ccs.ornl.gov kernel: [1928728.947628] Lustre: atlas2-OST0214: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4h6.ccs.ornl.gov kernel: [1928715.137632] Lustre: atlas2-OST0085: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4h6.ccs.ornl.gov kernel: [1928715.169362] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss4c4.ccs.ornl.gov kernel: [1928728.313814] Lustre: atlas2-OST029b: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3e6.ccs.ornl.gov kernel: [1928713.952432] Lustre: Skipped 1 previous similar message Jul 17 20:39:16 atlas-oss3c7.ccs.ornl.gov kernel: [1928717.249555] Lustre: atlas2-OST01c6: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1g3.ccs.ornl.gov kernel: [713358.246262] Lustre: atlas1-OST00c2: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3a8.ccs.ornl.gov kernel: [715025.066243] Lustre: atlas2-OST0127: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4d7.ccs.ornl.gov kernel: [1928717.098681] Lustre: atlas2-OST0216: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3c1.ccs.ornl.gov kernel: [1928718.517887] Lustre: atlas2-OST0130: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1h1.ccs.ornl.gov kernel: [793466.719940] Lustre: atlas1-OST00c8: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4b1.ccs.ornl.gov kernel: [1928720.893720] Lustre: atlas2-OST0170: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4f3.ccs.ornl.gov kernel: [1928711.040230] Lustre: atlas2-OST0102: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4g4.ccs.ornl.gov kernel: [1928730.913734] Lustre: atlas2-OST022b: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3g7.ccs.ornl.gov kernel: [1927313.944288] Lustre: atlas2-OST0276: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3f8.ccs.ornl.gov kernel: [815418.754692] Lustre: atlas2-OST026f: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3i6.ccs.ornl.gov kernel: [1928715.205311] Lustre: atlas2-OST0165: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3b6.ccs.ornl.gov kernel: [1928714.586252] Lustre: atlas2-OST012d: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1c2.ccs.ornl.gov kernel: [793464.629972] Lustre: atlas1-OST0011: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3e3.ccs.ornl.gov kernel: [1928713.033800] Lustre: atlas2-OST01d2: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4h1.ccs.ornl.gov kernel: [1928710.205504] Lustre: atlas2-OST02c0: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2e6.ccs.ornl.gov kernel: [786218.085036] Lustre: atlas1-OST006d: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3e8.ccs.ornl.gov kernel: [1928716.475299] Lustre: atlas2-OST0267: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1f8.ccs.ornl.gov kernel: [793469.464979] Lustre: atlas1-OST00bf: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4e6.ccs.ornl.gov kernel: [1928732.233439] Lustre: atlas2-OST033d: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss4i6.ccs.ornl.gov kernel: [1928729.840198] Lustre: atlas2-OST02cd: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss3i2.ccs.ornl.gov kernel: [1928722.336882] Lustre: atlas2-OST0311: Client 732728ee-4eb9-8f2f-4021-ad7af61279ba (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2b4.ccs.ornl.gov kernel: [786212.598238] Lustre: atlas1-OST0053: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1b8.ccs.ornl.gov kernel: [793462.125318] Lustre: atlas1-OST009f: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1g2.ccs.ornl.gov kernel: [713355.325192] Lustre: atlas1-OST00c1: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss1g4.ccs.ornl.gov kernel: [713354.329652] Lustre: atlas1-OST0153: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2c3.ccs.ornl.gov kernel: [786215.914654] Lustre: atlas1-OST005a: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2g1.ccs.ornl.gov kernel: [786214.802393] Lustre: atlas1-OST0078: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2a2.ccs.ornl.gov kernel: [786214.769535] Lustre: atlas1-OST0169: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2c7.ccs.ornl.gov kernel: [786217.479462] Lustre: atlas1-OST00ee: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2e4.ccs.ornl.gov kernel: [786217.269758] Lustre: atlas1-OST018b: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:39:16 atlas-oss2i2.ccs.ornl.gov kernel: [786210.650514] Lustre: Skipped 1 previous similar message Jul 17 20:39:54 atlas-mds3.ccs.ornl.gov kernel: [10571.034211] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 20:41:09 atlas-oss1h3.ccs.ornl.gov kernel: [793577.925916] LustreError: 137-5: atlas1-OST003e_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss1a3.ccs.ornl.gov kernel: [793577.072260] LustreError: 137-5: atlas1-OST0006_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss1i4.ccs.ornl.gov kernel: [793578.172404] LustreError: 137-5: atlas1-OST0167_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss2g2.ccs.ornl.gov kernel: [786325.253416] LustreError: 137-5: atlas1-OST010d_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss2g2.ccs.ornl.gov kernel: [786325.280047] LustreError: Skipped 6 previous similar messages Jul 17 20:41:09 atlas-oss2d6.ccs.ornl.gov kernel: [786325.420865] LustreError: 137-5: atlas1-OST0181_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss1e4.ccs.ornl.gov kernel: [793580.962062] LustreError: 137-5: atlas1-OST0027_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss2e1.ccs.ornl.gov kernel: [786333.418829] LustreError: 137-5: atlas1-OST018c_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss1b3.ccs.ornl.gov kernel: [793585.742616] LustreError: 137-5: atlas1-OST012e_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss1b3.ccs.ornl.gov kernel: [793585.770600] LustreError: Skipped 6 previous similar messages Jul 17 20:41:09 atlas-oss1a6.ccs.ornl.gov kernel: [793582.034736] LustreError: 137-5: atlas1-OST0001_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss1c6.ccs.ornl.gov kernel: [793575.380959] LustreError: 137-5: atlas1-OST0251_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss1h8.ccs.ornl.gov kernel: [793581.103252] LustreError: 137-5: atlas1-OST003b_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss2g7.ccs.ornl.gov kernel: [786321.532050] LustreError: 137-5: atlas1-OST022a_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss1c3.ccs.ornl.gov kernel: [793580.823572] LustreError: 137-5: atlas1-OST01c6_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss1c3.ccs.ornl.gov kernel: [793580.851715] LustreError: Skipped 6 previous similar messages Jul 17 20:41:09 atlas-oss2c2.ccs.ornl.gov kernel: [786325.437746] LustreError: 137-5: atlas1-OST00ed_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss2f5.ccs.ornl.gov kernel: [786331.157066] LustreError: 137-5: atlas1-OST0220_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss2c2.ccs.ornl.gov kernel: [786325.467920] LustreError: Skipped 6 previous similar messages Jul 17 20:41:09 atlas-oss2e6.ccs.ornl.gov kernel: [786331.672531] LustreError: 137-5: atlas1-OST02a9_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss1i6.ccs.ornl.gov kernel: [793580.074020] LustreError: 137-5: atlas1-OST0161_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss2c7.ccs.ornl.gov kernel: [786331.068399] LustreError: 137-5: atlas1-OST03ba_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss1e5.ccs.ornl.gov kernel: [793576.339799] LustreError: 137-5: atlas1-OST01d0_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss1d3.ccs.ornl.gov kernel: [793582.314444] LustreError: 137-5: atlas1-OST01ce_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss2e7.ccs.ornl.gov kernel: [786326.495189] LustreError: 137-5: atlas1-OST02aa_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss2d8.ccs.ornl.gov kernel: [786327.732290] LustreError: 137-5: atlas1-OST03c3_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss1d1.ccs.ornl.gov kernel: [793585.374126] LustreError: 137-5: atlas1-OST037c_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:09 atlas-oss1g1.ccs.ornl.gov kernel: [713468.866838] LustreError: 137-5: atlas1-OST0394_UUID: not available for connect from 10.38.144.4@o2ib4 (no target) Jul 17 20:41:15 atlas-mds3.ccs.ornl.gov kernel: [10652.259286] Lustre: atlas2-MDT0000: Denying connection for new client 90278dae-cc4e-d152-6f91-5950391fa5a3 (at 8098@gni100), waiting for all 20156 known clients (19237 recovered, 889 in progress, and 30 evicted) to recover in 3:56 Jul 17 20:41:15 atlas-mds3.ccs.ornl.gov kernel: [10652.282471] Lustre: Skipped 27 previous similar messages Jul 17 20:43:21 atlas-mds3.ccs.ornl.gov kernel: [10779.126800] Lustre: atlas2-MDT0000: Client 25eca314-2213-dba4-4e7c-ae1acb406793 (at 15@gni2) reconnecting, waiting for 20156 clients in recovery for 1:49 Jul 17 20:43:21 atlas-mds3.ccs.ornl.gov kernel: [10779.142392] Lustre: Skipped 370 previous similar messages Jul 17 20:43:21 atlas-mds3.ccs.ornl.gov kernel: [10779.148539] Lustre: atlas2-MDT0000: Client 25eca314-2213-dba4-4e7c-ae1acb406793 (at 15@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 20:43:21 atlas-mds3.ccs.ornl.gov kernel: [10779.163247] Lustre: Skipped 370 previous similar messages Jul 17 20:43:39 atlas-oss1a2.ccs.ornl.gov kernel: [793728.564351] Lustre: atlas1-OST0001: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss2g6.ccs.ornl.gov kernel: [786475.799984] Lustre: atlas1-OST010d: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss2c6.ccs.ornl.gov kernel: [786473.881262] Lustre: atlas1-OST00ed: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss1e8.ccs.ornl.gov kernel: [793730.694610] Lustre: atlas1-OST0027: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss1d7.ccs.ornl.gov kernel: [793732.556286] Lustre: atlas1-OST01ce: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss1a7.ccs.ornl.gov kernel: [793726.030019] Lustre: atlas1-OST0006: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss1c7.ccs.ornl.gov kernel: [793731.388081] Lustre: atlas1-OST01c6: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss2d2.ccs.ornl.gov kernel: [786478.007928] Lustre: atlas1-OST0181: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss1i8.ccs.ornl.gov kernel: [793730.158508] Lustre: atlas1-OST0167: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss1h4.ccs.ornl.gov kernel: [793730.894770] Lustre: atlas1-OST003b: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss1h7.ccs.ornl.gov kernel: [793731.812924] Lustre: atlas1-OST003e: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss2f1.ccs.ornl.gov kernel: [786428.099533] Lustre: atlas1-OST0220: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss1e1.ccs.ornl.gov kernel: [793735.310940] Lustre: atlas1-OST01d0: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss1b7.ccs.ornl.gov kernel: [793724.999660] Lustre: atlas1-OST012e: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss2e5.ccs.ornl.gov kernel: [786479.304061] Lustre: atlas1-OST018c: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss1i2.ccs.ornl.gov kernel: [793728.508033] Lustre: atlas1-OST0161: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss2g3.ccs.ornl.gov kernel: [786470.600573] Lustre: atlas1-OST022a: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:43:39 atlas-oss2g3.ccs.ornl.gov kernel: [786470.630993] Lustre: Skipped 1 previous similar message Jul 17 20:46:09 atlas-oss2e3.ccs.ornl.gov kernel: [785996.304478] Lustre: atlas1-OST02aa: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:46:09 atlas-oss1d5.ccs.ornl.gov kernel: [793881.767333] Lustre: atlas1-OST037c: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:46:09 atlas-oss1g5.ccs.ornl.gov kernel: [713773.160270] Lustre: atlas1-OST0394: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:46:09 atlas-oss2d4.ccs.ornl.gov kernel: [786574.451441] Lustre: atlas1-OST03c3: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:46:09 atlas-oss2c3.ccs.ornl.gov kernel: [786629.614616] Lustre: atlas1-OST03ba: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:46:09 atlas-oss1c2.ccs.ornl.gov kernel: [793878.326904] Lustre: atlas1-OST0251: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:46:09 atlas-oss2e2.ccs.ornl.gov kernel: [786628.280485] Lustre: atlas1-OST02a9: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:46:09 atlas-oss2g6.ccs.ornl.gov kernel: [786625.860693] Lustre: atlas1-OST03dd: Client ebe517ba-0354-d19f-ce11-70b366de9806 (at 10.38.144.4@o2ib4) reconnecting Jul 17 20:46:10 atlas-mds3.ccs.ornl.gov kernel: [10947.384769] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10095@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff881d03effb40/0xf35a0587ba9bcfe4 lrc: 3/0,0 mode: PR/PR res: [0x2000d23e1:0x5099:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x20 nid: 10095@gni100 remote: 0x115bd220a03082a6 expref: 131 pid: 15382 timeout: 4305602157 lvb_type: 0 used 0 Jul 17 20:46:10 atlas-mds3.ccs.ornl.gov kernel: [10947.426727] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 5 previous similar messages Jul 17 20:46:10 atlas-mds3.ccs.ornl.gov kernel: [10947.438203] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff881d06988240/0xf35a0587ba9898b5 lrc: 3/0,0 mode: PR/PR res: [0x2000d435e:0xe1f:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x20 nid: 8652@gni100 remote: 0x5abf0e67d084ed77 expref: 127 pid: 15491 timeout: 0 lvb_type: 0 used 0 Jul 17 20:46:10 atlas-mds3.ccs.ornl.gov kernel: [10947.476296] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) Skipped 3 previous similar messages Jul 17 20:51:20 atlas-mds3.ccs.ornl.gov kernel: [11257.896311] Lustre: atlas2-MDT0000: Denying connection for new client 5e3d9974-6429-e74f-5f56-dd2c35c42265 (at 11441@gni100), waiting for all 20156 known clients (19236 recovered, 889 in progress, and 31 evicted) to recover in -6:-09 Jul 17 20:51:20 atlas-mds3.ccs.ornl.gov kernel: [11257.919769] Lustre: Skipped 34 previous similar messages Jul 17 20:52:26 atlas-mds3.ccs.ornl.gov kernel: [11323.752764] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 20:53:22 atlas-mds3.ccs.ornl.gov kernel: [11379.843421] Lustre: atlas2-MDT0000: Client 533e6c21-1383-fda1-c45e-a0204e3de622 (at 34@gni2) reconnecting, waiting for 20156 clients in recovery for 4:20 Jul 17 20:53:22 atlas-mds3.ccs.ornl.gov kernel: [11379.859013] Lustre: Skipped 375 previous similar messages Jul 17 20:53:22 atlas-mds3.ccs.ornl.gov kernel: [11379.865163] Lustre: atlas2-MDT0000: Client 533e6c21-1383-fda1-c45e-a0204e3de622 (at 34@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 20:53:22 atlas-mds3.ccs.ornl.gov kernel: [11379.879873] Lustre: Skipped 375 previous similar messages Jul 17 21:01:32 atlas-mds3.ccs.ornl.gov kernel: [11870.489649] Lustre: atlas2-MDT0000: Denying connection for new client 122c7971-09a7-1195-19eb-ebdd8dbac0b0 (at 4287@gni100), waiting for all 20156 known clients (19252 recovered, 872 in progress, and 32 evicted) to recover in -3:-49 Jul 17 21:01:32 atlas-mds3.ccs.ornl.gov kernel: [11870.513040] Lustre: Skipped 35 previous similar messages Jul 17 21:02:27 atlas-mds3.ccs.ornl.gov kernel: [11925.321074] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 601s: evicting client at 5552@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff881d042eb000/0xf35a0587ba9b570f lrc: 3/0,0 mode: PR/PR res: [0x2000d2ada:0x1b3b:0x0].0 bits 0x13 rrc: 4 type: IBT flags: 0x20 nid: 5552@gni100 remote: 0xfedb3f6fafca61b expref: 129 pid: 15670 timeout: 4306579035 lvb_type: 0 used 0 Jul 17 21:02:27 atlas-mds3.ccs.ornl.gov kernel: [11925.362741] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 1 previous similar message Jul 17 21:02:27 atlas-mds3.ccs.ornl.gov kernel: [11925.375087] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 21:02:27 atlas-mds3.ccs.ornl.gov kernel: [11925.385216] Lustre: atlas2-MDT0000: disconnecting 1 stale clients Jul 17 21:02:27 atlas-mds3.ccs.ornl.gov kernel: [11925.527414] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff881d06d4a480/0xf35a0587ba98c188 lrc: 3/0,0 mode: PR/PR res: [0x2000d45ad:0xd70:0x0].0 bits 0x13 rrc: 18 type: IBT flags: 0x20 nid: 12516@gni100 remote: 0x7df460ed95762702 expref: 125 pid: 15712 timeout: 0 lvb_type: 0 used 0 Jul 17 21:02:27 atlas-mds3.ccs.ornl.gov kernel: [11925.565668] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) Skipped 2 previous similar messages Jul 17 21:03:22 atlas-mds3.ccs.ornl.gov kernel: [11980.471191] Lustre: atlas2-MDT0000: Client 23f5cb73-67a2-35a3-8292-da5854fe8410 (at 46@gni2) reconnecting, waiting for 20156 clients in recovery for 4:22 Jul 17 21:03:22 atlas-mds3.ccs.ornl.gov kernel: [11980.486783] Lustre: Skipped 371 previous similar messages Jul 17 21:03:22 atlas-mds3.ccs.ornl.gov kernel: [11980.492931] Lustre: atlas2-MDT0000: Client 23f5cb73-67a2-35a3-8292-da5854fe8410 (at 46@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 21:03:22 atlas-mds3.ccs.ornl.gov kernel: [11980.507633] Lustre: Skipped 371 previous similar messages Jul 17 21:04:26 atlas-oss1g5.ccs.ornl.gov kernel: [714870.048694] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0274_UUID lock: ffff880dfcf60240/0x40c6faaa64f4fb2c lrc: 3/0,0 mode: PR/PR res: [0x22f22c:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac843f0a expref: 658 pid: 18154 timeout: 5009266958 lvb_type: 1 Jul 17 21:04:26 atlas-oss1d5.ccs.ornl.gov kernel: [794978.690964] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST025c_UUID lock: ffff880ece737240/0x8dbaed8434697e6d lrc: 3/0,0 mode: PR/PR res: [0x24ae0d:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac845530 expref: 699 pid: 39837 timeout: 5089345920 lvb_type: 1 Jul 17 21:04:26 atlas-oss2g2.ccs.ornl.gov kernel: [787722.354603] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0079_UUID lock: ffff880de1200d80/0x83331598feb392be lrc: 3/0,0 mode: PR/PR res: [0x244b4c:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac843f18 expref: 691 pid: 113754 timeout: 5082094855 lvb_type: 1 Jul 17 21:04:26 atlas-oss2a3.ccs.ornl.gov kernel: [787729.507586] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST01fa_UUID lock: ffff88094b156480/0x70099c07d523bcdd lrc: 3/0,0 mode: PR/PR res: [0x243608:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac8456bf expref: 728 pid: 116339 timeout: 5082096803 lvb_type: 1 Jul 17 21:04:26 atlas-oss1c3.ccs.ornl.gov kernel: [794978.028151] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST00a2_UUID lock: ffff880e6088dd80/0xefd0ac4302b21b8a lrc: 3/0,0 mode: PR/PR res: [0x22f655:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac84481e expref: 609 pid: 93695 timeout: 5089344755 lvb_type: 1 Jul 17 21:04:26 atlas-oss1d4.ccs.ornl.gov kernel: [794975.050256] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST013b_UUID lock: ffff880af33df240/0xb6190dbaa3356d lrc: 3/0,0 mode: PR/PR res: [0x247ce9:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac844825 expref: 583 pid: 38781 timeout: 5089343878 lvb_type: 1 Jul 17 21:04:26 atlas-oss1d4.ccs.ornl.gov kernel: [794975.181863] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 1 previous similar message Jul 17 21:04:26 atlas-oss1d4.ccs.ornl.gov kernel: [794975.212210] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST025b_UUID lock: ffff880b8068b000/0xb6190dbaa335e4 lrc: 3/0,0 mode: PR/PR res: [0x245ff5:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac84653d expref: 760 pid: 38781 timeout: 5089343878 lvb_type: 1 Jul 17 21:04:26 atlas-oss1a4.ccs.ornl.gov kernel: [794967.520386] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0123_UUID lock: ffff880e8d9b0000/0xbf3dfea6743d5372 lrc: 3/0,0 mode: PR/PR res: [0x237298:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac843f26 expref: 626 pid: 94180 timeout: 5089342630 lvb_type: 1 Jul 17 21:04:26 atlas-oss1b4.ccs.ornl.gov kernel: [794978.789758] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST000b_UUID lock: ffff880913717480/0xd18237f829dc6f20 lrc: 3/0,0 mode: PR/PR res: [0x243dac:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac844833 expref: 661 pid: 38369 timeout: 5089344615 lvb_type: 1 Jul 17 21:04:26 atlas-oss1c8.ccs.ornl.gov kernel: [794977.748446] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0137_UUID lock: ffff8809667fe000/0x2ffe4f2f20098f1 lrc: 3/0,0 mode: PR/PR res: [0x243e46:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac843f03 expref: 658 pid: 38873 timeout: 5089344618 lvb_type: 1 Jul 17 21:04:26 atlas-oss2h1.ccs.ornl.gov kernel: [787726.266047] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0230_UUID lock: ffff880de76e2b40/0x725566934aea4b7e lrc: 3/0,0 mode: PR/PR res: [0x23c8ef:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac84551b expref: 668 pid: 113893 timeout: 5082095608 lvb_type: 1 Jul 17 21:04:26 atlas-oss2h1.ccs.ornl.gov kernel: [787726.392239] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0110_UUID lock: ffff880b3b607900/0x725566934aea4af9 lrc: 3/0,0 mode: PR/PR res: [0x23cb27:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac843ef5 expref: 697 pid: 113835 timeout: 5082095612 lvb_type: 1 Jul 17 21:04:26 atlas-oss2f3.ccs.ornl.gov kernel: [787720.913345] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST02b2_UUID lock: ffff8809349ed240/0x378011ecfcb186e0 lrc: 3/0,0 mode: PR/PR res: [0x241271:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac846544 expref: 686 pid: 22096 timeout: 5082096600 lvb_type: 1 Jul 17 21:04:26 atlas-oss2f3.ccs.ornl.gov kernel: [787721.040688] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST02b2_UUID lock: ffff880fa1c77900/0x378011ecfcb18677 lrc: 3/0,0 mode: PR/PR res: [0x24123c:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac843efc expref: 686 pid: 22096 timeout: 5082096604 lvb_type: 1 Jul 17 21:04:26 atlas-oss2b7.ccs.ornl.gov kernel: [787726.551425] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST00e6_UUID lock: ffff880da4f396c0/0xe6565b6beca4d9a3 lrc: 3/0,0 mode: PR/PR res: [0x24c07e:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac845529 expref: 722 pid: 118351 timeout: 5082096433 lvb_type: 1 Jul 17 21:04:26 atlas-oss2b7.ccs.ornl.gov kernel: [787726.679847] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0176_UUID lock: ffff880918e6a480/0xe6565b6beca4d9aa lrc: 3/0,0 mode: PR/PR res: [0x244a7d:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac8456cd expref: 664 pid: 118351 timeout: 5082096433 lvb_type: 1 Jul 17 21:04:27 atlas-oss1f3.ccs.ornl.gov kernel: [794978.412099] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST026a_UUID lock: ffff880ce219b000/0x2986e3331ae03f97 lrc: 3/0,0 mode: PR/PR res: [0x249f8d:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac8456c6 expref: 711 pid: 38793 timeout: 5089344310 lvb_type: 1 Jul 17 21:04:27 atlas-oss1h6.ccs.ornl.gov kernel: [794979.732451] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST030d_UUID lock: ffff880bfd801000/0xc55818efcf8992a4 lrc: 3/0,0 mode: PR/PR res: [0x237d26:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac843f1f expref: 691 pid: 91609 timeout: 5089346251 lvb_type: 1 Jul 17 21:04:27 atlas-oss2c7.ccs.ornl.gov kernel: [787728.834877] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST03be_UUID lock: ffff880993167000/0x8e9696610ce87a84 lrc: 3/0,0 mode: PR/PR res: [0x24da6b:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac84482c expref: 697 pid: 122856 timeout: 5082098189 lvb_type: 1 Jul 17 21:04:27 atlas-oss2f5.ccs.ornl.gov kernel: [787728.956812] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST02b4_UUID lock: ffff880f7ef1f000/0x73b9bd9f0c840f03 lrc: 3/0,0 mode: PR/PR res: [0x246de6:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac845522 expref: 686 pid: 118182 timeout: 5082098159 lvb_type: 1 Jul 17 21:04:27 atlas-oss2f5.ccs.ornl.gov kernel: [787729.095319] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 2 previous similar messages Jul 17 21:04:27 atlas-oss2a4.ccs.ornl.gov kernel: [787727.983768] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST031b_UUID lock: ffff880ba78246c0/0x31a79a96c8cbe731 lrc: 3/0,0 mode: PR/PR res: [0x2459ba:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x10020 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac84654b expref: 704 pid: 117396 timeout: 5082098055 lvb_type: 1 Jul 17 21:08:43 atlas-mds3.ccs.ornl.gov kernel: [12301.684108] Lustre: atlas2-MDT0000: disconnecting 1 stale clients Jul 17 21:11:38 atlas-mds3.ccs.ornl.gov kernel: [12476.929104] Lustre: atlas2-MDT0000: Denying connection for new client 03d88ac4-e14f-f281-32b2-28f78a6bc8b7 (at 11625@gni100), waiting for all 20156 known clients (19408 recovered, 710 in progress, and 38 evicted) to recover in 2:22 Jul 17 21:11:38 atlas-mds3.ccs.ornl.gov kernel: [12476.952379] Lustre: Skipped 35 previous similar messages Jul 17 21:13:22 atlas-mds3.ccs.ornl.gov kernel: [12581.835053] Lustre: atlas2-MDT0000: Client 79e3fa41-2b6c-d3d2-20fa-65e6085a41fa (at 45@gni2) reconnecting, waiting for 20156 clients in recovery for 0:37 Jul 17 21:13:22 atlas-mds3.ccs.ornl.gov kernel: [12581.847243] Lustre: atlas2-MDT0000: Client fcdec0da-ab38-143f-5820-bf15f4f0e989 (at 33@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 21:13:22 atlas-mds3.ccs.ornl.gov kernel: [12581.847245] Lustre: Skipped 375 previous similar messages Jul 17 21:13:22 atlas-mds3.ccs.ornl.gov kernel: [12581.871504] Lustre: Skipped 377 previous similar messages Jul 17 21:14:59 atlas-mds3.ccs.ornl.gov kernel: [12678.041423] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 8710@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff881d05cd16c0/0xf35a0587ba9a56b6 lrc: 3/0,0 mode: PR/PR res: [0x2000d2c2f:0xbb9:0x0].0 bits 0x13 rrc: 10 type: IBT flags: 0x20 nid: 8710@gni100 remote: 0x5fdeea460d6eb0d8 expref: 131 pid: 15286 timeout: 4307331107 lvb_type: 0 used 0 Jul 17 21:14:59 atlas-mds3.ccs.ornl.gov kernel: [12678.083177] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 3 previous similar messages Jul 17 21:14:59 atlas-mds3.ccs.ornl.gov kernel: [12678.095704] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 21:14:59 atlas-mds3.ccs.ornl.gov kernel: [12678.104030] Lustre: Skipped 1 previous similar message Jul 17 21:14:59 atlas-mds3.ccs.ornl.gov kernel: [12678.162019] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff883fc5f87240/0xf35a0587ba9e60e1 lrc: 3/0,0 mode: PR/PR res: [0x2000d1f90:0x2fa5:0x0].0 bits 0x13 rrc: 7 type: IBT flags: 0x20 nid: 5627@gni100 remote: 0xb6e0cd513f327a55 expref: 130 pid: 15794 timeout: 0 lvb_type: 0 used 0 Jul 17 21:14:59 atlas-mds3.ccs.ornl.gov kernel: [12678.200188] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) Skipped 3 previous similar messages Jul 17 21:21:59 atlas-mds3.ccs.ornl.gov kernel: [13098.564934] Lustre: atlas2-MDT0000: Denying connection for new client 25be746c-2550-85d6-1e29-7ba4a135d231 (at 5559@gni100), waiting for all 20156 known clients (19468 recovered, 648 in progress, and 40 evicted) to recover in -1:-43 Jul 17 21:21:59 atlas-mds3.ccs.ornl.gov kernel: [13098.588309] Lustre: Skipped 40 previous similar messages Jul 17 21:23:22 atlas-mds3.ccs.ornl.gov kernel: [13182.436495] Lustre: atlas2-MDT0000: Client fcdec0da-ab38-143f-5820-bf15f4f0e989 (at 33@gni2) reconnecting, waiting for 20156 clients in recovery for -3:-06 Jul 17 21:23:22 atlas-mds3.ccs.ornl.gov kernel: [13182.437032] Lustre: atlas2-MDT0000: Client 56767aa3-5bee-5531-3f7d-5cdc84d3035e (at 35@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 21:23:22 atlas-mds3.ccs.ornl.gov kernel: [13182.437035] Lustre: Skipped 370 previous similar messages Jul 17 21:23:22 atlas-mds3.ccs.ornl.gov kernel: [13182.473101] Lustre: Skipped 370 previous similar messages Jul 17 21:24:24 atlas-mds3.ccs.ornl.gov kernel: [13243.595696] Lustre: atlas2-MDT0000: disconnecting 1 stale clients Jul 17 21:30:40 atlas-mds3.ccs.ornl.gov kernel: [13619.943096] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 4284@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff883f3e9feb40/0xf35a0587ba9c9efe lrc: 3/0,0 mode: PR/PR res: [0x2000d2f1b:0x2063:0x0].0 bits 0x13 rrc: 16 type: IBT flags: 0x20 nid: 4284@gni100 remote: 0x64d195532765c546 expref: 131 pid: 15859 timeout: 4308272025 lvb_type: 0 used 0 Jul 17 21:30:40 atlas-mds3.ccs.ornl.gov kernel: [13619.984930] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 2 previous similar messages Jul 17 21:30:40 atlas-mds3.ccs.ornl.gov kernel: [13619.997612] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 21:30:40 atlas-mds3.ccs.ornl.gov kernel: [13620.005968] Lustre: Skipped 1 previous similar message Jul 17 21:30:40 atlas-mds3.ccs.ornl.gov kernel: [13620.016890] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff883fc39b2d80/0xf35a0587ba99320d lrc: 3/0,0 mode: PR/PR res: [0x2000d1f90:0x30f7:0x0].0 bits 0x13 rrc: 463 type: IBT flags: 0x20 nid: 7114@gni100 remote: 0xcd622bd195dd6e89 expref: 130 pid: 15717 timeout: 0 lvb_type: 0 used 0 Jul 17 21:30:40 atlas-mds3.ccs.ornl.gov kernel: [13620.055257] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) Skipped 3 previous similar messages Jul 17 21:32:03 atlas-mds3.ccs.ornl.gov kernel: [13703.355386] Lustre: atlas2-MDT0000: Denying connection for new client 7c9cecb7-6c21-5cab-c1b6-5ab153ff6158 (at 8173@gni100), waiting for all 20156 known clients (19488 recovered, 623 in progress, and 45 evicted) to recover in 3:53 Jul 17 21:32:03 atlas-mds3.ccs.ornl.gov kernel: [13703.378569] Lustre: Skipped 44 previous similar messages Jul 17 21:33:22 atlas-mds3.ccs.ornl.gov kernel: [13783.020628] Lustre: atlas2-MDT0000: Client 0eae4b5d-f757-e2fd-a32e-aebade19d063 (at 3@gni2) reconnecting, waiting for 20156 clients in recovery for 2:34 Jul 17 21:33:22 atlas-mds3.ccs.ornl.gov kernel: [13783.036115] Lustre: Skipped 367 previous similar messages Jul 17 21:33:22 atlas-mds3.ccs.ornl.gov kernel: [13783.042287] Lustre: atlas2-MDT0000: Client 0eae4b5d-f757-e2fd-a32e-aebade19d063 (at 3@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 21:33:22 atlas-mds3.ccs.ornl.gov kernel: [13783.056933] Lustre: Skipped 371 previous similar messages Jul 17 21:35:14 atlas-oss2c3.ccs.ornl.gov kernel: [789574.825794] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST005a_UUID lock: ffff880dda0d0480/0x950cd86563b124cb lrc: 3/0,0 mode: PR/PR res: [0x56590d:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac8832d7 expref: 728 pid: 122409 timeout: 5083945899 lvb_type: 1 Jul 17 21:36:56 atlas-mds3.ccs.ornl.gov kernel: [13996.313946] Lustre: atlas2-MDT0000: disconnecting 1 stale clients Jul 17 21:40:36 atlas-oss2i4.ccs.ornl.gov kernel: [789892.600714] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST035b_UUID lock: ffff880999c8e000/0xd740bef4fd5af0cb lrc: 3/0,0 mode: PR/PR res: [0x589ced:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac883531 expref: 670 pid: 118297 timeout: 5084266905 lvb_type: 1 Jul 17 21:40:37 atlas-oss2g6.ccs.ornl.gov kernel: [789894.899324] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST02bd_UUID lock: ffff880ee43e7d80/0x5e7f571017bb1ae4 lrc: 3/0,0 mode: PR/PR res: [0x50a788:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac8834cf expref: 600 pid: 114037 timeout: 5084267976 lvb_type: 1 Jul 17 21:40:37 atlas-oss2g6.ccs.ornl.gov kernel: [789895.026407] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 2 previous similar messages Jul 17 21:40:42 atlas-oss2b4.ccs.ornl.gov kernel: [789900.111260] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0203_UUID lock: ffff880d682e3900/0xb6b2b6227133b233 lrc: 3/0,0 mode: PR/PR res: [0x554433:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac883394 expref: 590 pid: 115134 timeout: 5084273932 lvb_type: 1 Jul 17 21:40:53 atlas-oss1e8.ccs.ornl.gov kernel: [797165.814278] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST0147_UUID lock: ffff880e5cfe9900/0x24931d398ab24eb6 lrc: 3/0,0 mode: PR/PR res: [0x50970c:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac883260 expref: 700 pid: 37674 timeout: 5091531910 lvb_type: 1 Jul 17 21:40:58 atlas-oss1d8.ccs.ornl.gov kernel: [797169.985185] LustreError: 21:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.38.144.4@o2ib4 ns: filter-atlas1-OST001f_UUID lock: ffff880a6a681480/0x8e87152ef9cecd56 lrc: 3/0,0 mode: PR/PR res: [0x523800:0x0:0x0].0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x20 nid: 10.38.144.4@o2ib4 remote: 0xb01ca1e8ac8833e8 expref: 657 pid: 38630 timeout: 5091537218 lvb_type: 1 Jul 17 21:42:32 atlas-mds3.ccs.ornl.gov kernel: [14332.744365] Lustre: atlas2-MDT0000: Denying connection for new client 12bc7b97-d600-a5e8-25d5-e48bbca5becb (at 7186@gni100), waiting for all 20156 known clients (19572 recovered, 535 in progress, and 49 evicted) to recover in 0:-18 Jul 17 21:42:32 atlas-mds3.ccs.ornl.gov kernel: [14332.767612] Lustre: Skipped 49 previous similar messages Jul 17 21:43:22 atlas-mds3.ccs.ornl.gov kernel: [14383.620939] Lustre: atlas2-MDT0000: Client 8c34c9cb-7f47-6a1e-e2df-6a82cffa74f8 (at 8@gni2) reconnecting, waiting for 20156 clients in recovery for -1:-09 Jul 17 21:43:22 atlas-mds3.ccs.ornl.gov kernel: [14383.636634] Lustre: Skipped 374 previous similar messages Jul 17 21:43:23 atlas-mds3.ccs.ornl.gov kernel: [14383.642840] Lustre: atlas2-MDT0000: Client 8c34c9cb-7f47-6a1e-e2df-6a82cffa74f8 (at 8@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 21:43:23 atlas-mds3.ccs.ornl.gov kernel: [14383.657487] Lustre: Skipped 373 previous similar messages Jul 17 21:46:57 atlas-mds3.ccs.ornl.gov kernel: [14597.879265] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 601s: evicting client at 3801@gni100 ns: mdt-atlas2-MDT0000_UUID lock: ffff881d03c4d6c0/0xf35a0587ba9b7515 lrc: 3/0,0 mode: PR/PR res: [0x2000d3958:0xadd:0x0].0 bits 0x13 rrc: 6 type: IBT flags: 0x20 nid: 3801@gni100 remote: 0x50306ab2b3fefec6 expref: 131 pid: 15420 timeout: 4309249139 lvb_type: 0 used 0 Jul 17 21:46:57 atlas-mds3.ccs.ornl.gov kernel: [14597.920921] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 5 previous similar messages Jul 17 21:46:57 atlas-mds3.ccs.ornl.gov kernel: [14597.941666] Lustre: atlas2-MDT0000: recovery is timed out, evict stale exports Jul 17 21:46:57 atlas-mds3.ccs.ornl.gov kernel: [14597.950007] Lustre: Skipped 1 previous similar message Jul 17 21:46:57 atlas-mds3.ccs.ornl.gov kernel: [14598.029876] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) ### BUG 6063: lock collide during recovery ns: mdt-atlas2-MDT0000_UUID lock: ffff88400530ab40/0xf35a0587ba984430 lrc: 3/0,0 mode: PR/PR res: [0x2000d1fbf:0x1f9b:0x0].0 bits 0x13 rrc: 13 type: IBT flags: 0x20 nid: 7198@gni100 remote: 0xc28c7310a04f4009 expref: 131 pid: 15891 timeout: 0 lvb_type: 0 used 0 Jul 17 21:46:57 atlas-mds3.ccs.ornl.gov kernel: [14598.070040] LustreError: 19309:0:(ldlm_lockd.c:878:ldlm_server_blocking_ast()) Skipped 3 previous similar messages Jul 17 21:52:44 atlas-mds3.ccs.ornl.gov kernel: [14945.337736] Lustre: atlas2-MDT0000: Denying connection for new client a1fb3595-7001-0dda-321f-5fd13b8c4588 (at 3791@gni100), waiting for all 20156 known clients (19624 recovered, 482 in progress, and 50 evicted) to recover in 0:-29 Jul 17 21:52:44 atlas-mds3.ccs.ornl.gov kernel: [14945.361015] Lustre: Skipped 56 previous similar messages Jul 17 21:53:23 atlas-mds3.ccs.ornl.gov kernel: [14984.251765] Lustre: atlas2-MDT0000: Client d9d1d25e-e5ed-dd11-1f9e-8e7ee06a20f0 (at 2@gni2) reconnecting, waiting for 20156 clients in recovery for -1:-08 Jul 17 21:53:23 atlas-mds3.ccs.ornl.gov kernel: [14984.267456] Lustre: Skipped 372 previous similar messages Jul 17 21:53:23 atlas-mds3.ccs.ornl.gov kernel: [14984.273604] Lustre: atlas2-MDT0000: Client d9d1d25e-e5ed-dd11-1f9e-8e7ee06a20f0 (at 2@gni2) refused reconnection, still busy with 1 active RPCs Jul 17 21:53:23 atlas-mds3.ccs.ornl.gov kernel: [14984.288217] Lustre: Skipped 374 previous similar messages Jul 17 21:56:22 atlas-mds3.ccs.ornl.gov kernel: [15163.423277] Lustre: atlas2-MDT0000: disconnecting 1 stale clients Jul 17 21:56:22 atlas-mds3.ccs.ornl.gov kernel: [15163.555549] LustreError: 19309:0:(mdt_lvb.c:157:mdt_lvbo_fill()) atlas2-MDT0000: expected 128 actual 0. Jul 17 21:56:22 atlas-mds3.ccs.ornl.gov kernel: [15163.582419] LustreError: 19309:0:(mdt_lvb.c:157:mdt_lvbo_fill()) atlas2-MDT0000: expected 128 actual 0. Jul 17 21:58:17 atlas-mds3.ccs.ornl.gov kernel: [15278.957331] Lustre: atlas2-MDT0000: Client 1c0dca9e-5706-aca6-e344-f6aef05ad87b (at 14@gni2) reconnecting Jul 17 21:58:17 atlas-mds3.ccs.ornl.gov kernel: [15278.961384] Lustre: atlas2-MDT0000: Client 27762a33-afc6-af9d-7ab0-b0429d1d85e5 (at 16@gni2) reconnecting Jul 17 21:58:18 atlas-mds3.ccs.ornl.gov kernel: [15279.718961] Lustre: atlas2-MDT0000: Client 1d186399-bf63-fee6-2b63-8fddd9e7fba3 (at 83@gni2) reconnecting Jul 17 21:58:18 atlas-mds3.ccs.ornl.gov kernel: [15279.729876] Lustre: Skipped 1 previous similar message Jul 17 21:58:18 atlas-mds3.ccs.ornl.gov kernel: [15279.932809] Lustre: 19309:0:(ldlm_lib.c:2094:target_recovery_thread()) too long recovery - read logs Jul 17 21:58:18 atlas-mds3.ccs.ornl.gov kernel: [15279.943331] LustreError: dumping log to /tmp/lustre-log.1405648698.19309 Jul 17 21:58:18 atlas-oss3h7.ccs.ornl.gov kernel: [1933467.011086] LustreError: 71903:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Jul 17 21:58:18 atlas-mds3.ccs.ornl.gov kernel: [15279.981554] Lustre: atlas2-MDT0000: Recovery over after 245:11, of 20156 clients 20104 recovered and 52 were evicted. Jul 17 21:58:18 atlas-oss4b8.ccs.ornl.gov kernel: [1933462.185542] LustreError: 15662:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Jul 17 21:58:18 atlas-oss4h8.ccs.ornl.gov kernel: [1933451.269863] LustreError: 84277:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Jul 17 21:58:18 atlas-oss3h1.ccs.ornl.gov kernel: [1933449.021990] LustreError: 61500:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Jul 17 21:58:18 atlas-oss3d7.ccs.ornl.gov kernel: [1933461.391959] LustreError: 16833:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Jul 17 21:58:21 atlas-mds3.ccs.ornl.gov kernel: [15283.438195] Lustre: atlas2-MDT0000: Client 25eca314-2213-dba4-4e7c-ae1acb406793 (at 15@gni2) reconnecting Jul 17 21:58:24 atlas-mds3.ccs.ornl.gov kernel: [15285.515302] Lustre: atlas2-MDT0000: Client f6e8db44-3d15-27c2-bda3-864f4bb0dd2f (at 95@gni2) reconnecting Jul 17 21:58:24 atlas-mds3.ccs.ornl.gov kernel: [15285.526223] Lustre: Skipped 36 previous similar messages Jul 17 21:58:36 atlas-mds3.ccs.ornl.gov kernel: [15298.133548] Lustre: atlas2-MDT0000: Client 4f110c1a-4217-694e-2203-96aa8971e180 (at 10.36.202.138@o2ib) reconnecting Jul 17 21:58:36 atlas-mds3.ccs.ornl.gov kernel: [15298.145534] Lustre: Skipped 37 previous similar messages Jul 17 21:59:29 atlas-mds3.ccs.ornl.gov kernel: [15350.684914] Lustre: atlas2-MDT0000: Client 88af36eb-fc99-9aa3-c80e-6b143a6ac03a (at 10.36.225.5@o2ib) reconnecting Jul 17 21:59:29 atlas-mds3.ccs.ornl.gov kernel: [15350.696707] Lustre: Skipped 4 previous similar messages Jul 17 21:59:51 atlas-mds3.ccs.ornl.gov kernel: [15373.536267] Lustre: atlas2-MDT0000: Client 1841c050-d33f-3b1f-84ac-2fe38e66471e (at 10.36.202.175@o2ib) reconnecting Jul 17 22:00:46 atlas-mds3.ccs.ornl.gov kernel: [15427.786867] Lustre: atlas2-MDT0000: Client 84eb7626-264a-9db5-eff3-ee82928c6490 (at 44@gni2) reconnecting Jul 17 22:00:46 atlas-mds3.ccs.ornl.gov kernel: [15427.797783] Lustre: Skipped 4 previous similar messages Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.381954] INFO: task mdt00_000:15203 blocked for more than 120 seconds. Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.389665] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.398633] mdt00_000 D 0000000000000002 0 15203 2 0x00000000 Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.406483] ffff881e85227560 0000000000000046 ffff880000001000 00000000e028a510 Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.415032] ffff884051f26440 04c88840d1000000 ffff881e6a06a980 10000000000004c8 Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.423606] ffff88205015fab8 ffff881e85227fd8 000000000000fb88 ffff88205015fab8 Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.432161] Call Trace: Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.435008] [] ? kobject_put+0x27/0x60 Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.441161] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.448474] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.455112] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.462252] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.469886] [] ? down_write+0x32/0x40 Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.475981] [] lod_alloc_qos.clone.0+0x175/0x1180 [lod] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.483819] [] ? qsd_op_begin+0x5f/0xb40 [lquota] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.491046] [] lod_qos_prep_create+0x74a/0x1b14 [lod] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.498675] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.506117] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.514461] [] ? osd_xattr_get+0x21b/0x2d0 [osd_ldiskfs] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.522374] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.530511] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.539511] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.546842] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.554487] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.561269] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.569122] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.576366] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.584811] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.592927] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.599699] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.607141] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.614413] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.621774] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.629424] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.637467] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.644012] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.651447] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.659756] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.667020] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.675847] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.682999] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.690743] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:01:56 atlas-mds3.ccs.ornl.gov kernel: [15498.698478] [] ? default_wake_function+0x0/0x20 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.705548] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.712718] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.719882] [] child_rip+0xa/0x20 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.725563] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.732711] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.739844] [] ? child_rip+0x0/0x20 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.745707] INFO: task mdt00_101:15304 blocked for more than 120 seconds. Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.753410] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.762373] mdt00_101 D 000000000000000c 0 15304 2 0x00000000 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.770210] ffff881e82835560 0000000000000046 ffff881e76bcf7b0 ffff881ebb09b338 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.778764] ffff884044857678 ffff884044823e30 ffff881e357d8198 ffff883fd3657408 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.787344] ffff881e82833ab8 ffff881e82835fd8 000000000000fb88 ffff881e82833ab8 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.795914] Call Trace: Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.798757] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.806074] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.812709] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.819831] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.827440] [] ? down_write+0x32/0x40 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.833501] [] lod_alloc_qos.clone.0+0x175/0x1180 [lod] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.841328] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.847969] [] ? qsd_op_begin+0x5f/0xb40 [lquota] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.855205] [] lod_qos_prep_create+0x74a/0x1b14 [lod] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.862837] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.870259] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.878575] [] ? osd_xattr_get+0x21b/0x2d0 [osd_ldiskfs] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.886479] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.894579] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.903556] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.910921] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.918542] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.925281] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.933091] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.940357] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.948775] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.956881] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.963621] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.971041] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.978280] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.985618] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15498.993245] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.001253] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.007827] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.015297] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.023640] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.030906] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.039698] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.046826] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.054557] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.062267] [] ? __wake_up+0x53/0x70 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.068237] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.075383] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.082510] [] child_rip+0xa/0x20 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.088196] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.095340] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.102463] [] ? child_rip+0x0/0x20 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.108325] INFO: task mdt00_128:15331 blocked for more than 120 seconds. Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.116031] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.125001] mdt00_128 D 0000000000000000 0 15331 2 0x00000000 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.132845] ffff881e824c9560 0000000000000046 ffff881e824c9520 0000000000000286 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.141389] ffff881e824c94d0 ffff881dc12ecbc0 ffff881e824c9520 ffff881dc12ecbc0 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.149932] ffff881e824c7af8 ffff881e824c9fd8 000000000000fb88 ffff881e824c7af8 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.158479] Call Trace: Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.161332] [] ? cfs_free+0xe/0x10 [libcfs] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.167964] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.175278] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.181919] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.190959] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.198568] [] ? down_write+0x32/0x40 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.204630] [] lod_alloc_qos.clone.0+0x175/0x1180 [lod] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.212434] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.219079] [] ? qsd_op_begin+0x5f/0xb40 [lquota] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.226303] [] lod_qos_prep_create+0x74a/0x1b14 [lod] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.233918] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.241337] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.249636] [] ? osd_xattr_get+0x21b/0x2d0 [osd_ldiskfs] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.257540] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.265640] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.274616] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.281942] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.289552] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.296296] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.304105] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.311337] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.319754] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.327858] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.334602] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.342027] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.349254] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.356595] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.364227] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.372237] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.378784] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.386229] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.394529] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.401782] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.410565] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.417694] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.425429] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.433137] [] ? __wake_up+0x53/0x70 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.439119] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.446271] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.453397] [] child_rip+0xa/0x20 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.459086] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.466236] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.473360] [] ? child_rip+0x0/0x20 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.479235] INFO: task mdt00_181:15384 blocked for more than 120 seconds. Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.486947] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.495913] mdt00_181 D 0000000000000004 0 15384 2 0x00000000 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.503769] ffff881e81dcd560 0000000000000046 ffff881e81dcd528 ffff881e81dcd524 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.512345] ffff881e81dcd4d0 ffff88207fc28e00 ffff88011c416740 0000000000000400 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.520903] ffff881e81d7c5f8 ffff881e81dcdfd8 000000000000fb88 ffff881e81d7c5f8 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.529458] Call Trace: Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.532302] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.539623] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.546261] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.553390] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.560997] [] ? down_write+0x32/0x40 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.567061] [] lod_alloc_qos.clone.0+0x175/0x1180 [lod] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.574859] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.581496] [] ? qsd_op_begin+0x5f/0xb40 [lquota] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.588719] [] lod_qos_prep_create+0x74a/0x1b14 [lod] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.596334] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.603754] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.612054] [] ? osd_xattr_get+0x21b/0x2d0 [osd_ldiskfs] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.619958] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.628063] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.637045] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.644363] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.651976] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.658714] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.666530] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.673760] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.682184] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.690287] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.697029] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:01:57 atlas-mds3.ccs.ornl.gov kernel: [15499.704454] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.711680] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.719022] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.726655] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.734661] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.741208] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.748652] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.756950] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.764199] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.772983] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.780114] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.787840] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.795574] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.802719] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.809846] [] child_rip+0xa/0x20 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.815531] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.822681] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.829888] [] ? child_rip+0x0/0x20 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.835747] INFO: task mdt00_186:15389 blocked for more than 120 seconds. Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.843449] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.852417] mdt00_186 D 0000000000000000 0 15389 2 0x00000000 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.860258] ffff881e81ead560 0000000000000046 ffff881e81ead610 0000000000000000 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.868808] ffff883ff4ca04a0 0000000004c91d74 ffff881e81ead5dc 00000000811b5c6c Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.877356] ffff881e81e65058 ffff881e81eadfd8 000000000000fb88 ffff881e81e65058 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.885900] Call Trace: Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.888744] [] ? ldiskfs_ind_get_blocks+0xc4/0x610 [ldiskfs] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.897132] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.904449] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.911089] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.918203] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.925815] [] ? down_write+0x32/0x40 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.931873] [] lod_alloc_qos.clone.0+0x175/0x1180 [lod] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.939680] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.946323] [] ? qsd_op_begin+0x5f/0xb40 [lquota] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.953546] [] lod_qos_prep_create+0x74a/0x1b14 [lod] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.961158] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.968566] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.976863] [] ? osd_xattr_get+0x21b/0x2d0 [osd_ldiskfs] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.984770] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15499.992870] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.001847] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.009172] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.016788] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.023527] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.031342] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.038572] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.046989] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.055093] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.061833] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.069256] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.076484] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.083825] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.091460] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.099465] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.106012] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.113449] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.121748] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.128988] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.137773] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.144903] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.152639] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.160349] [] ? default_wake_function+0x0/0x20 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.167401] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.174547] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.181670] [] child_rip+0xa/0x20 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.187359] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.194517] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.201641] [] ? child_rip+0x0/0x20 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.207510] INFO: task mdt00_230:15433 blocked for more than 120 seconds. Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.215207] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.224182] mdt00_230 D 0000000000000000 0 15433 2 0x00000000 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.232026] ffff881e81615560 0000000000000046 0000000000000002 ffff880000034d60 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.240574] ffff881e81613540 00000000000c1250 ffff881e81615610 ffffffff8112be33 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.249134] ffff881e81613af8 ffff881e81615fd8 000000000000fb88 ffff881e81613af8 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.257673] Call Trace: Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.260519] [] ? __alloc_pages_nodemask+0x113/0x8d0 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.267934] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.275249] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.282373] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.289979] [] ? down_write+0x32/0x40 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.296038] [] lod_alloc_qos.clone.0+0x175/0x1180 [lod] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.303850] [] ? qsd_op_begin+0x5f/0xb40 [lquota] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.311075] [] lod_qos_prep_create+0x74a/0x1b14 [lod] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.320680] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.328100] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.336395] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.344489] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.353474] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.360795] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.368410] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.375150] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.382965] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.390197] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.398611] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.406720] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.413469] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.420897] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.428125] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.435478] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.443145] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.451151] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.457697] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.465136] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.473426] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.480670] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.489461] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.496590] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.504324] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.512038] [] ? __wake_up+0x53/0x70 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.518006] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.525151] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.532274] [] child_rip+0xa/0x20 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.537958] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.545103] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.552225] [] ? child_rip+0x0/0x20 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.558087] INFO: task mdt00_239:15442 blocked for more than 120 seconds. Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.565790] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.574760] mdt00_239 D 0000000000000000 0 15442 2 0x00000000 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.582600] ffff881e81791560 0000000000000046 0000000000000000 0000000000000000 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.591154] ffff883ff4ca04a0 ffff881e04c91d74 ffff881e817915dc 0000000081791540 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.599707] ffff881e8178fab8 ffff881e81791fd8 000000000000fb88 ffff881e8178fab8 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.608257] Call Trace: Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.611101] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.618427] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.625074] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.632211] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.639827] [] ? down_write+0x32/0x40 Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.645900] [] lod_alloc_qos.clone.0+0x175/0x1180 [lod] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.653725] [] ? qsd_op_begin+0x5f/0xb40 [lquota] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.660957] [] lod_qos_prep_create+0x74a/0x1b14 [lod] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.668575] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.675998] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.684302] [] ? osd_xattr_get+0x21b/0x2d0 [osd_ldiskfs] Jul 17 22:01:58 atlas-mds3.ccs.ornl.gov kernel: [15500.692207] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.700319] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.709330] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.716672] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.724312] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.731074] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.738919] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.746178] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.754635] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.762756] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.769520] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.776970] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.784220] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.791594] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.799358] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.807392] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.813959] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.821422] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.829726] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.836972] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.845764] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.852899] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.860636] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.868351] [] ? __wake_up+0x53/0x70 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.874335] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.881481] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.888608] [] child_rip+0xa/0x20 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.894293] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.901444] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.908559] [] ? child_rip+0x0/0x20 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.914421] INFO: task mdt00_247:15450 blocked for more than 120 seconds. Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.922131] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.931102] mdt00_247 D 000000000000000c 0 15450 2 0x00000000 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.938940] ffff881e810e5560 0000000000000046 0000000000000002 ffff880000034d60 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.947498] ffff881e81096040 00000000000c1250 ffff881e810e5610 ffffffff8112be33 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.956039] ffff881e810965f8 ffff881e810e5fd8 000000000000fb88 ffff881e810965f8 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.964577] Call Trace: Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.967425] [] ? __alloc_pages_nodemask+0x113/0x8d0 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.974843] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.982163] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.988798] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15500.995921] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.003529] [] ? down_write+0x32/0x40 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.009593] [] lod_alloc_qos.clone.0+0x175/0x1180 [lod] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.017412] [] ? qsd_op_begin+0x5f/0xb40 [lquota] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.024637] [] lod_qos_prep_create+0x74a/0x1b14 [lod] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.032251] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.039671] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.047974] [] ? osd_xattr_get+0x21b/0x2d0 [osd_ldiskfs] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.055873] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.063982] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.072959] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.080283] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.087899] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.094639] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.102453] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.109686] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.118102] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.126207] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.132951] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.140376] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.147607] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.154944] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.162577] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.170573] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.177123] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.184565] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.192869] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.200121] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.208904] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.216037] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.223765] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.231474] [] ? __wake_up+0x53/0x70 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.237453] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.244605] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.251730] [] child_rip+0xa/0x20 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.257418] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.264563] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.271686] [] ? child_rip+0x0/0x20 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.277548] INFO: task mdt01_017:15476 blocked for more than 120 seconds. Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.285250] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.294221] mdt01_017 D 0000000000000006 0 15476 2 0x00000000 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.302060] ffff881e80d75560 0000000000000046 ffff881e80d75520 0000000000000286 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.310610] ffff881e80d754d0 ffff881dce1936c0 ffff881e80d75520 ffff881dce1936c0 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.319251] ffff881e80d2d098 ffff881e80d75fd8 000000000000fb88 ffff881e80d2d098 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.327805] Call Trace: Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.330657] [] ? cfs_free+0xe/0x10 [libcfs] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.337295] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.344612] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.351250] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.358375] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.365984] [] ? down_write+0x32/0x40 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.372043] [] lod_alloc_qos.clone.0+0x175/0x1180 [lod] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.379856] [] ? qsd_op_begin+0x5f/0xb40 [lquota] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.387073] [] lod_qos_prep_create+0x74a/0x1b14 [lod] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.394683] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.402103] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.410401] [] ? osd_xattr_get+0x21b/0x2d0 [osd_ldiskfs] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.418311] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.426406] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.435384] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.444654] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.452267] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.459007] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.466824] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.474055] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.482471] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.490579] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.497317] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.504736] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.511960] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.519290] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.526919] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.534928] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.541478] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.548922] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.557223] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.564471] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.573260] [] ? default_wake_function+0x12/0x20 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.580391] [] ? __wake_up_common+0x59/0x90 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.587034] [] ? __wake_up+0x53/0x70 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.593025] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.600178] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.607303] [] child_rip+0xa/0x20 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.612994] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.620140] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.627265] [] ? child_rip+0x0/0x20 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.633129] INFO: task mdt01_048:15507 blocked for more than 120 seconds. Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.640829] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.649805] mdt01_048 D 0000000000000012 0 15507 2 0x00000000 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.657645] ffff881e80aa3560 0000000000000046 ffff881e80aa3610 0000000000000000 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.666207] ffff881e5192f760 ffff881e04c91d90 ffff881e80aa35dc 0000000080aa3540 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.674755] ffff881e80a52638 ffff881e80aa3fd8 000000000000fb88 ffff881e80a52638 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.683310] Call Trace: Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.686168] [] ? ldiskfs_ind_get_blocks+0xc4/0x610 [ldiskfs] Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.694557] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.701197] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:01:59 atlas-mds3.ccs.ornl.gov kernel: [15501.708508] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.715148] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.722271] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.729884] [] ? down_write+0x32/0x40 Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.735946] [] lod_alloc_qos.clone.0+0x175/0x1180 [lod] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.743752] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.750399] [] ? qsd_op_begin+0x5f/0xb40 [lquota] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.757624] [] lod_qos_prep_create+0x74a/0x1b14 [lod] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.765235] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.772660] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.780969] [] ? osd_xattr_get+0x21b/0x2d0 [osd_ldiskfs] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.788884] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.796985] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.805966] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.813290] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.820904] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.827648] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.835470] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.842709] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.851135] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.859239] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.865984] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.873416] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.880651] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.887997] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.895632] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.903642] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.910192] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.917641] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.925927] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.933184] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.941983] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.949128] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.956881] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.964604] [] ? default_wake_function+0x0/0x20 Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.971668] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.978819] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.985949] [] child_rip+0xa/0x20 Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.991658] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15501.998823] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:02:00 atlas-mds3.ccs.ornl.gov kernel: [15502.005957] [] ? child_rip+0x0/0x20 Jul 17 22:02:21 atlas-mds3.ccs.ornl.gov kernel: [15523.676058] Lustre: atlas2-MDT0000: Client f04d019e-f170-2481-7488-6181a0e28e3e (at 10.36.202.176@o2ib) reconnecting Jul 17 22:02:21 atlas-mds3.ccs.ornl.gov kernel: [15523.688126] Lustre: Skipped 3 previous similar messages Jul 17 22:08:16 atlas-mds3.ccs.ornl.gov kernel: [15878.739785] Lustre: atlas2-OST0244-osc-MDT0000: slow creates, last=[0x0:0x1:0x0], next=[0x0:0x1:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0, status=0 Jul 17 22:09:05 atlas-mds3.ccs.ornl.gov kernel: [15927.354319] Lustre: 15781:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-49), not sending early reply Jul 17 22:09:05 atlas-mds3.ccs.ornl.gov kernel: [15927.354321] req@ffff883fd6404400 x1473082053370021/t0(0) o101->fbe63d19-2146-6b38-6a44-76b38fc6ceaa@9311@gni100:0/0 lens 568/3384 e 1 to 0 dl 1405649350 ref 2 fl Interpret:/0/0 rc 0/0 Jul 17 22:09:05 atlas-mds3.ccs.ornl.gov kernel: [15927.355391] Lustre: 15377:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-49), not sending early reply Jul 17 22:09:05 atlas-mds3.ccs.ornl.gov kernel: [15927.355392] req@ffff881e51e29800 x1473082054454745/t0(0) o101->7e50cc6a-398c-db1c-84aa-71d3c4c39743@94@gni100:0/0 lens 1024/3512 e 1 to 0 dl 1405649350 ref 2 fl Interpret:/0/0 rc 0/0 Jul 17 22:09:05 atlas-mds3.ccs.ornl.gov kernel: [15927.418921] Lustre: 15781:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 14 previous similar messages Jul 17 22:09:05 atlas-mds3.ccs.ornl.gov kernel: [15927.919914] Lustre: 15555:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-49), not sending early reply Jul 17 22:09:05 atlas-mds3.ccs.ornl.gov kernel: [15927.919915] req@ffff881e76539000 x1473082066145304/t0(0) o101->1c53c3ef-11b0-f60f-ac99-2c97e382ee3f@12@gni100:0/0 lens 552/3384 e 1 to 0 dl 1405649350 ref 2 fl Interpret:/0/0 rc 0/0 Jul 17 22:09:05 atlas-mds3.ccs.ornl.gov kernel: [15927.952046] Lustre: 15555:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 22 previous similar messages Jul 17 22:09:07 atlas-mds3.ccs.ornl.gov kernel: [15929.356232] Lustre: 15884:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-49), not sending early reply Jul 17 22:09:07 atlas-mds3.ccs.ornl.gov kernel: [15929.356234] req@ffff881e5a5e2800 x1473185547472573/t0(0) o101->ced5a9bf-ed12-ec09-e41b-c604e72bea77@454@gni3:0/0 lens 656/3512 e 1 to 0 dl 1405649352 ref 2 fl Interpret:/0/0 rc 0/0 Jul 17 22:09:07 atlas-mds3.ccs.ornl.gov kernel: [15929.388303] Lustre: 15884:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 18 previous similar messages Jul 17 22:09:12 atlas-mds3.ccs.ornl.gov kernel: [15934.362017] Lustre: 15826:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-49), not sending early reply Jul 17 22:09:12 atlas-mds3.ccs.ornl.gov kernel: [15934.362019] req@ffff883f57d8b000 x1473185597566095/t0(0) o101->b0ff3a62-f9fb-76a1-1da6-153c611e1017@150@gni3:0/0 lens 664/3512 e 1 to 0 dl 1405649357 ref 2 fl Interpret:/0/0 rc 0/0 Jul 17 22:09:12 atlas-mds3.ccs.ornl.gov kernel: [15934.394059] Lustre: 15826:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 12 previous similar messages Jul 17 22:09:40 atlas-mds3.ccs.ornl.gov kernel: [15962.771934] Lustre: atlas2-MDT0000: Client fdf957bd-4457-0034-ecb5-687b2414db45 (at 31@gni2) reconnecting Jul 17 22:09:40 atlas-mds3.ccs.ornl.gov kernel: [15962.782858] Lustre: atlas2-MDT0000: Client fdf957bd-4457-0034-ecb5-687b2414db45 (at 31@gni2) refused reconnection, still busy with 3 active RPCs Jul 17 22:09:40 atlas-mds3.ccs.ornl.gov kernel: [15962.797562] Lustre: Skipped 172 previous similar messages Jul 17 22:10:18 atlas-mds3.ccs.ornl.gov kernel: [16000.989771] Lustre: 15525:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-49), not sending early reply Jul 17 22:10:18 atlas-mds3.ccs.ornl.gov kernel: [16000.989773] req@ffff881e5efc9800 x1472638697609264/t0(0) o101->88af36eb-fc99-9aa3-c80e-6b143a6ac03a@10.36.225.5@o2ib:0/0 lens 584/3384 e 1 to 0 dl 1405649423 ref 2 fl Interpret:/0/0 rc 0/0 Jul 17 22:10:18 atlas-mds3.ccs.ornl.gov kernel: [16001.022577] Lustre: 15525:0:(service.c:1339:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages Jul 17 22:10:46 atlas-mds3.ccs.ornl.gov kernel: [16028.736482] Lustre: lock timed out (enqueued at 1405648696, 750s ago) Jul 17 22:10:46 atlas-mds3.ccs.ornl.gov kernel: [16028.738545] Lustre: lock timed out (enqueued at 1405648696, 750s ago) Jul 17 22:10:46 atlas-mds3.ccs.ornl.gov kernel: [16028.738614] LustreError: dumping log to /tmp/lustre-log.1405649446.16039 Jul 17 22:10:46 atlas-mds3.ccs.ornl.gov kernel: [16028.758724] Lustre: Skipped 1 previous similar message Jul 17 22:10:48 atlas-mds3.ccs.ornl.gov kernel: [16030.397045] Lustre: lock timed out (enqueued at 1405648698, 750s ago) Jul 17 22:10:48 atlas-mds3.ccs.ornl.gov kernel: [16030.404378] Lustre: Skipped 23 previous similar messages Jul 17 22:11:59 atlas-mds3.ccs.ornl.gov kernel: [16102.087703] Lustre: lock timed out (enqueued at 1405648769, 750s ago) Jul 17 22:11:59 atlas-mds3.ccs.ornl.gov kernel: [16102.095017] Lustre: Skipped 9 previous similar messages Jul 17 22:14:40 atlas-mds3.ccs.ornl.gov kernel: [16263.071665] Lustre: atlas2-MDT0000: Client fdf957bd-4457-0034-ecb5-687b2414db45 (at 31@gni2) reconnecting Jul 17 22:14:40 atlas-mds3.ccs.ornl.gov kernel: [16263.082579] Lustre: Skipped 4 previous similar messages Jul 17 22:17:04 atlas-mds3.ccs.ornl.gov kernel: [16407.161477] Lustre: lock timed out (enqueued at 1405648696, 1128s ago) Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.166343] LNet: Service thread pid 15384 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.185740] Pid: 15384, comm: mdt00_181 Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.190133] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.190133] Call Trace: Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.194739] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.202059] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.208693] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.215817] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.223423] [] ? down_write+0x32/0x40 Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.229482] [] lod_alloc_qos.clone.0+0x175/0x1180 [lod] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.237280] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.243926] [] ? qsd_op_begin+0x5f/0xb40 [lquota] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.251147] [] lod_qos_prep_create+0x74a/0x1b14 [lod] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.258762] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.266179] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.274464] [] ? osd_xattr_get+0x21b/0x2d0 [osd_ldiskfs] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.282365] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.290456] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.299427] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.306740] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.314350] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.321085] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.328908] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.331483] Lustre: atlas2-OST0245-osc-MDT0000: slow creates, last=[0x0:0x1:0x0], next=[0x0:0x1:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0, status=0 Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.354400] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.362827] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.370935] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.377705] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.385126] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.392428] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.399764] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.407397] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.415412] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.421959] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.429406] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.437697] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.444935] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.453714] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.460838] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.468575] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.476308] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.483463] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.490588] [] child_rip+0xa/0x20 Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.496288] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.503436] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.510559] [] ? child_rip+0x0/0x20 Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.516414] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.518183] LustreError: dumping log to /tmp/lustre-log.1405649896.15384 Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.605092] LNet: Service thread pid 15433 was inactive for 1200.16s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.624475] Pid: 15433, comm: mdt00_230 Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.628873] Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.628874] Call Trace: Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.633474] [] ? __alloc_pages_nodemask+0x113/0x8d0 Jul 17 22:18:16 atlas-mds3.ccs.ornl.gov kernel: [16479.640892] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.648210] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.655332] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.662937] [] ? down_write+0x32/0x40 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.669002] [] lod_alloc_qos.clone.0+0x175/0x1180 [lod] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.676819] [] ? qsd_op_begin+0x5f/0xb40 [lquota] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.684036] [] lod_qos_prep_create+0x74a/0x1b14 [lod] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.691640] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.699058] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.707341] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.715441] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.724414] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.731733] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.739342] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.746080] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.753895] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.761127] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.769532] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.777627] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.784366] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.791780] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.799007] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.806336] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.813967] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.821968] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.828501] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.835923] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.844222] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.851460] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.860235] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.867354] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.875081] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.882788] [] ? __wake_up+0x53/0x70 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.888762] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.895904] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.903023] [] child_rip+0xa/0x20 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.908694] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.915832] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.922952] [] ? child_rip+0x0/0x20 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.928808] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.930577] Pid: 15450, comm: mdt00_247 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.934970] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.934970] Call Trace: Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.939574] [] ? __alloc_pages_nodemask+0x113/0x8d0 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.946986] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.954300] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.960932] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.968062] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.975668] [] ? down_write+0x32/0x40 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.981722] [] lod_alloc_qos.clone.0+0x175/0x1180 [lod] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.989520] [] ? qsd_op_begin+0x5f/0xb40 [lquota] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16479.996742] [] lod_qos_prep_create+0x74a/0x1b14 [lod] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.004347] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.011761] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.020043] [] ? osd_xattr_get+0x21b/0x2d0 [osd_ldiskfs] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.027948] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.036046] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.045014] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.052340] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.059949] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.066685] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.074495] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.081720] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.090125] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.098226] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.104958] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.112376] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.119593] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.126925] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.134548] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.142553] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.149096] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.156531] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.164821] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.172058] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.180834] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.187950] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.195675] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.203379] [] ? __wake_up+0x53/0x70 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.209353] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.216495] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.223612] [] child_rip+0xa/0x20 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.229302] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.236454] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.243574] [] ? child_rip+0x0/0x20 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.249427] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.251199] LNet: Service thread pid 15509 was inactive for 1200.02s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.270571] LNet: Skipped 1 previous similar message Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.276230] Pid: 15509, comm: mdt01_050 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.280623] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.280624] Call Trace: Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.285228] [] ? perf_event_task_sched_out+0x33/0x80 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.292736] [] ? load_balance_fair+0x208/0x2f0 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.299662] [] ? __switch_to+0x1ac/0x320 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.306005] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.313319] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.319952] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.327071] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.334673] [] ? down_write+0x32/0x40 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.340721] [] lod_alloc_qos.clone.0+0x175/0x1180 [lod] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.348521] [] ? sync_buffer+0x0/0x50 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.354581] [] ? qsd_op_begin+0x5f/0xb40 [lquota] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.361798] [] lod_qos_prep_create+0x74a/0x1b14 [lod] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.369410] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.376819] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.385113] [] ? osd_xattr_get+0x21b/0x2d0 [osd_ldiskfs] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.393000] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.401100] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.410069] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.417381] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.424990] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.431730] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.439554] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.446882] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.455296] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.463406] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.470149] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.477569] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.484796] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.492140] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.499768] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.507767] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.514317] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.521763] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.530065] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.537311] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.548097] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.555224] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.562949] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.570658] [] ? __wake_up+0x53/0x70 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.576637] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.583794] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.590921] [] child_rip+0xa/0x20 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.596625] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.603767] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.610888] [] ? child_rip+0x0/0x20 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.616736] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.618507] Pid: 16089, comm: mdt03_118 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.622902] Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.622902] Call Trace: Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.627504] [] ? __alloc_pages_nodemask+0x113/0x8d0 Jul 17 22:18:17 atlas-mds3.ccs.ornl.gov kernel: [16480.634915] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.642232] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.648874] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.656013] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.663624] [] ? down_write+0x32/0x40 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.669678] [] lod_alloc_qos.clone.0+0x175/0x1180 [lod] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.677487] [] ? qsd_op_begin+0x5f/0xb40 [lquota] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.684705] [] lod_qos_prep_create+0x74a/0x1b14 [lod] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.692312] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.699728] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.708021] [] ? osd_xattr_get+0x21b/0x2d0 [osd_ldiskfs] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.715921] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.724011] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.732982] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.740302] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.747903] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.754640] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.762447] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.769665] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.778078] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.786176] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.792911] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.800328] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.807550] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.814880] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.822507] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.830506] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.837054] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.844486] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.852777] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.860107] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.868886] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.876009] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.883732] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.891437] [] ? __wake_up+0x53/0x70 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.897409] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.904546] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.911669] [] child_rip+0xa/0x20 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.917348] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.924487] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.931601] [] ? child_rip+0x0/0x20 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.937453] Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.939222] LNet: Service thread pid 15476 was inactive for 1200.01s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.953916] LNet: Service thread pid 15521 was inactive for 1200.02s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16480.989247] LustreError: dumping log to /tmp/lustre-log.1405649898.15564 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16481.074340] LustreError: dumping log to /tmp/lustre-log.1405649898.16069 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16481.085335] LustreError: dumping log to /tmp/lustre-log.1405649898.15507 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16481.096381] LustreError: dumping log to /tmp/lustre-log.1405649898.16185 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16481.118241] LustreError: dumping log to /tmp/lustre-log.1405649898.15602 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16481.181256] LustreError: dumping log to /tmp/lustre-log.1405649898.15791 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16481.214417] LustreError: dumping log to /tmp/lustre-log.1405649898.15371 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16481.268370] LustreError: dumping log to /tmp/lustre-log.1405649898.15382 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16481.306437] LustreError: dumping log to /tmp/lustre-log.1405649898.16101 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16481.478682] LNet: Service thread pid 15229 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16481.493396] LNet: Skipped 60 previous similar messages Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16481.499250] LustreError: dumping log to /tmp/lustre-log.1405649898.15229 Jul 17 22:18:18 atlas-mds3.ccs.ornl.gov kernel: [16481.562753] LustreError: dumping log to /tmp/lustre-log.1405649898.15212 Jul 17 22:18:19 atlas-mds3.ccs.ornl.gov kernel: [16482.478711] LustreError: dumping log to /tmp/lustre-log.1405649899.15560 Jul 17 22:18:23 atlas-mds3.ccs.ornl.gov kernel: [16485.720683] LNet: Service thread pid 16102 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jul 17 22:18:23 atlas-mds3.ccs.ornl.gov kernel: [16485.735393] LNet: Skipped 2 previous similar messages Jul 17 22:18:23 atlas-mds3.ccs.ornl.gov kernel: [16485.741151] LustreError: dumping log to /tmp/lustre-log.1405649903.16102 Jul 17 22:18:23 atlas-mds3.ccs.ornl.gov kernel: [16485.849913] LustreError: dumping log to /tmp/lustre-log.1405649903.16000 Jul 17 22:18:23 atlas-mds3.ccs.ornl.gov kernel: [16486.150051] LustreError: dumping log to /tmp/lustre-log.1405649903.15933 Jul 17 22:18:24 atlas-mds3.ccs.ornl.gov kernel: [16486.829662] LustreError: dumping log to /tmp/lustre-log.1405649904.15767 Jul 17 22:18:24 atlas-mds3.ccs.ornl.gov kernel: [16487.310119] LustreError: dumping log to /tmp/lustre-log.1405649904.15442 Jul 17 22:19:29 atlas-mds3.ccs.ornl.gov kernel: [16552.518512] LNet: Service thread pid 15676 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Jul 17 22:19:29 atlas-mds3.ccs.ornl.gov kernel: [16552.533228] LNet: Skipped 4 previous similar messages Jul 17 22:19:29 atlas-mds3.ccs.ornl.gov kernel: [16552.538987] LustreError: dumping log to /tmp/lustre-log.1405649969.15676 Jul 17 22:19:40 atlas-mds3.ccs.ornl.gov kernel: [16563.360996] Lustre: atlas2-MDT0000: Client fdf957bd-4457-0034-ecb5-687b2414db45 (at 31@gni2) refused reconnection, still busy with 3 active RPCs Jul 17 22:19:40 atlas-mds3.ccs.ornl.gov kernel: [16563.375701] Lustre: Skipped 84 previous similar messages Jul 17 22:20:41 atlas-mds3.ccs.ornl.gov kernel: [16623.897828] Lustre: 15296:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-422), not sending early reply Jul 17 22:20:41 atlas-mds3.ccs.ornl.gov kernel: [16623.897830] req@ffff881e6cd90800 x1470544415695556/t0(0) o101->69cba22c-7e75-5451-e406-135326642f28@10.36.205.206@o2ib:0/0 lens 1024/3512 e 1 to 0 dl 1405650046 ref 2 fl Interpret:/0/0 rc 0/0 Jul 17 22:23:01 atlas-mds3.ccs.ornl.gov kernel: [16764.331301] Lustre: 15616:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Jul 17 22:23:01 atlas-mds3.ccs.ornl.gov kernel: [16764.331304] req@ffff881d047f9c00 x1472638697665212/t0(0) o101->88af36eb-fc99-9aa3-c80e-6b143a6ac03a@10.36.225.5@o2ib:0/0 lens 648/3512 e 0 to 0 dl 1405650186 ref 2 fl Interpret:/0/0 rc 0/0 Jul 17 22:23:14 atlas-mds3.ccs.ornl.gov kernel: [16776.964302] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.36.205.208@o2ib ns: mdt-atlas2-MDT0000_UUID lock: ffff881e7ae126c0/0xf35a0587ba3b7e25 lrc: 3/0,0 mode: PR/PR res: [0x2000d4ffb:0x2554:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x20 nid: 10.36.205.208@o2ib remote: 0x8bf1ff17f106e7de expref: 453656 pid: 15567 timeout: 4311426239 lvb_type: 0 used 0 Jul 17 22:23:14 atlas-mds3.ccs.ornl.gov kernel: [16777.007737] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) Skipped 1 previous similar message Jul 17 22:23:21 atlas-mds3.ccs.ornl.gov kernel: [16784.743163] Lustre: atlas2-MDT0000: Client 88af36eb-fc99-9aa3-c80e-6b143a6ac03a (at 10.36.225.5@o2ib) reconnecting Jul 17 22:23:21 atlas-mds3.ccs.ornl.gov kernel: [16784.754957] Lustre: Skipped 105 previous similar messages Jul 17 22:23:39 atlas-mds3.ccs.ornl.gov kernel: [16802.867290] LNet: Service thread pid 15415 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 17 22:23:39 atlas-mds3.ccs.ornl.gov kernel: [16802.886686] LNet: Skipped 1 previous similar message Jul 17 22:23:39 atlas-mds3.ccs.ornl.gov kernel: [16802.892362] Pid: 15415, comm: mdt00_212 Jul 17 22:23:39 atlas-mds3.ccs.ornl.gov kernel: [16802.896768] Jul 17 22:23:39 atlas-mds3.ccs.ornl.gov kernel: [16802.896769] Call Trace: Jul 17 22:23:39 atlas-mds3.ccs.ornl.gov kernel: [16802.901387] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:23:39 atlas-mds3.ccs.ornl.gov kernel: [16802.908033] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:23:39 atlas-mds3.ccs.ornl.gov kernel: [16802.915355] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:23:39 atlas-mds3.ccs.ornl.gov kernel: [16802.922466] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:23:39 atlas-mds3.ccs.ornl.gov kernel: [16802.930077] [] ? down_write+0x32/0x40 Jul 17 22:23:39 atlas-mds3.ccs.ornl.gov kernel: [16802.936136] [] lod_qos_prep_create+0xdca/0x1b14 [lod] Jul 17 22:23:39 atlas-mds3.ccs.ornl.gov kernel: [16802.943749] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:23:39 atlas-mds3.ccs.ornl.gov kernel: [16802.951164] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16802.959452] [] ? osd_xattr_get+0x21b/0x2d0 [osd_ldiskfs] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16802.967351] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16802.975455] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16802.984416] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16802.991741] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16802.999351] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.006087] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.013904] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.021136] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.029556] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.037656] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.044394] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.051809] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.059035] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.066385] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.074010] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.082017] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.088562] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.095996] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.104290] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.111526] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.120312] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.127438] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.135166] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.142871] [] ? __wake_up+0x53/0x70 Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.148846] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.155986] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.163108] [] child_rip+0xa/0x20 Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.168790] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.175931] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.183052] [] ? child_rip+0x0/0x20 Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.188911] Jul 17 22:23:40 atlas-mds3.ccs.ornl.gov kernel: [16803.190774] LustreError: dumping log to /tmp/lustre-log.1405650220.15415 Jul 17 22:24:41 atlas-mds3.ccs.ornl.gov kernel: [16864.388139] Lustre: 16224:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Jul 17 22:24:41 atlas-mds3.ccs.ornl.gov kernel: [16864.388141] req@ffff883f3ddc5400 x1471794962268630/t0(0) o101->6fe796dd-e0c1-ec87-7114-84eba0fb390e@10.36.205.199@o2ib:0/0 lens 544/3384 e 0 to 0 dl 1405650286 ref 2 fl Interpret:/0/0 rc 0/0 Jul 17 22:24:41 atlas-mds3.ccs.ornl.gov kernel: [16864.535167] Lustre: lock timed out (enqueued at 1405649531, 750s ago) Jul 17 22:24:41 atlas-mds3.ccs.ornl.gov kernel: [16864.542491] Lustre: Skipped 2 previous similar messages Jul 17 22:28:16 atlas-mds3.ccs.ornl.gov kernel: [17079.906133] Lustre: atlas2-OST0246-osc-MDT0000: slow creates, last=[0x0:0x1:0x0], next=[0x0:0x1:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0, status=0 Jul 17 22:29:51 atlas-mds3.ccs.ornl.gov kernel: [17175.001679] Lustre: atlas2-MDT0000: Client a3c87f05-1b27-9db1-a6f0-8fec93053578 (at 41@gni3) refused reconnection, still busy with 1 active RPCs Jul 17 22:29:51 atlas-mds3.ccs.ornl.gov kernel: [17175.016397] Lustre: Skipped 105 previous similar messages Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.875372] LNet: Service thread pid 15527 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.894750] Pid: 15527, comm: mdt01_068 Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.899155] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.899156] Call Trace: Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.903776] [] ? bh_lru_install+0x16e/0x1a0 Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.910436] [] rwsem_down_failed_common+0x95/0x1d0 Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.917778] [] rwsem_down_write_failed+0x23/0x30 Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.924921] [] call_rwsem_down_write_failed+0x13/0x20 Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.932547] [] ? down_write+0x32/0x40 Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.938637] [] lod_qos_prep_create+0xdca/0x1b14 [lod] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.946261] [] ? fld_server_lookup+0x72/0x3d0 [fld] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.953706] [] lod_declare_striped_object+0x14b/0x880 [lod] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.962031] [] ? osd_xattr_get+0x21b/0x2d0 [osd_ldiskfs] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.969956] [] lod_declare_object_create+0x511/0x7a0 [lod] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.978077] [] mdd_declare_object_create_internal+0xbf/0x1f0 [mdd] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.987093] [] mdd_declare_create+0x4e/0x870 [mdd] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17214.994429] [] ? mdd_linkea_prepare+0x23f/0x430 [mdd] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.002064] [] mdd_create+0x7c5/0x1790 [mdd] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.008815] [] ? osd_xattr_get+0x97/0x2d0 [osd_ldiskfs] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.016654] [] mdt_reint_open+0x13ae/0x21c0 [mdt] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.023903] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.032365] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.040473] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.047232] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.054673] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.061917] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.069269] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.076918] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.084940] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.091484] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.098902] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.106144] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.114923] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.122046] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.129774] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.137477] [] ? __wake_up+0x53/0x70 Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.143454] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.150689] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.157812] [] child_rip+0xa/0x20 Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.163556] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.170696] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.177817] [] ? child_rip+0x0/0x20 Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.183670] Jul 17 22:30:31 atlas-mds3.ccs.ornl.gov kernel: [17215.185430] LustreError: dumping log to /tmp/lustre-log.1405650631.15527 Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17314.966184] LNet: Service thread pid 15987 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17314.985578] Pid: 15987, comm: mdt03_016 Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17314.989995] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17314.989996] Call Trace: Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17314.994647] [] ? libcfs_debug_msg+0x41/0x50 [libcfs] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.002180] [] cfs_waitq_wait+0xe/0x10 [libcfs] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.009260] [] ldlm_completion_ast+0x57a/0x960 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.017077] [] ? ldlm_expired_completion_wait+0x0/0x390 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.025881] [] ? default_wake_function+0x0/0x20 Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.032943] [] ldlm_cli_enqueue_local+0x1f8/0x5d0 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.041075] [] ? ldlm_completion_ast+0x0/0x960 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.048911] [] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.056154] [] mdt_object_lock0+0x339/0xaf0 [mdt] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.063393] [] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.070651] [] ? ldlm_completion_ast+0x0/0x960 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.078482] [] mdt_object_lock+0x14/0x20 [mdt] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.085436] [] mdt_getattr_name_lock+0xe19/0x1980 [mdt] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.093283] [] ? lustre_msg_buf+0x55/0x60 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.100712] [] ? __req_capsule_get+0x166/0x700 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.108559] [] ? lustre_msg_get_flags+0x34/0xb0 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.116459] [] mdt_intent_getattr+0x2a2/0x4b0 [mdt] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.123903] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.131255] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.138903] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.146929] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.153492] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.160944] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.169250] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.176508] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.185296] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.192426] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.200157] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.207861] [] ? __wake_up+0x53/0x70 Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.213846] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.220989] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.228104] [] child_rip+0xa/0x20 Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.233805] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.240954] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.248079] [] ? child_rip+0x0/0x20 Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.253935] Jul 17 22:32:11 atlas-mds3.ccs.ornl.gov kernel: [17315.255705] LustreError: dumping log to /tmp/lustre-log.1405650731.15987 Jul 17 22:33:34 atlas-mds3.ccs.ornl.gov kernel: [17398.047597] Lustre: atlas2-MDT0000: Client d3e74f17-b36f-b930-b7ab-74eccaaa8cf1 (at 9307@gni100) reconnecting Jul 17 22:33:34 atlas-mds3.ccs.ornl.gov kernel: [17398.058905] Lustre: Skipped 118 previous similar messages Jul 17 22:37:03 atlas-mds3.ccs.ornl.gov kernel: [17607.168793] Lustre: lock timed out (enqueued at 1405650273, 750s ago) Jul 17 22:37:03 atlas-mds3.ccs.ornl.gov kernel: [17607.273949] Lustre: 15694:0:(service.c:1339:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply Jul 17 22:37:03 atlas-mds3.ccs.ornl.gov kernel: [17607.273951] req@ffff881d00dcbc00 x1473069879114838/t0(0) o101->3f642f0c-cc49-82f9-a487-89452748d3b1@10.36.205.208@o2ib:0/0 lens 1024/3512 e 0 to 0 dl 1405651028 ref 2 fl Interpret:/0/0 rc 0/0 Jul 17 22:38:16 atlas-mds3.ccs.ornl.gov kernel: [17680.499795] Lustre: atlas2-OST0247-osc-MDT0000: slow creates, last=[0x0:0x1:0x0], next=[0x0:0x1:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0, status=0 Jul 17 22:39:52 atlas-mds3.ccs.ornl.gov kernel: [17776.441161] Lustre: atlas2-MDT0000: Client 584aff53-0b51-8760-596e-826a018d64a8 (at 70@gni3) refused reconnection, still busy with 1 active RPCs Jul 17 22:39:52 atlas-mds3.ccs.ornl.gov kernel: [17776.455870] Lustre: Skipped 107 previous similar messages Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.596848] LNet: Service thread pid 15525 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.616229] Pid: 15525, comm: mdt01_066 Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.620633] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.620634] Call Trace: Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.625274] [] ? libcfs_debug_msg+0x41/0x50 [libcfs] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.632805] [] cfs_waitq_wait+0xe/0x10 [libcfs] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.639887] [] ldlm_completion_ast+0x57a/0x960 [ptlrpc] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.647721] [] ? ldlm_expired_completion_wait+0x0/0x390 [ptlrpc] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.656504] [] ? default_wake_function+0x0/0x20 Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.663570] [] ldlm_cli_enqueue_local+0x1f8/0x5d0 [ptlrpc] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.671693] [] ? ldlm_completion_ast+0x0/0x960 [ptlrpc] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.679523] [] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.686761] [] mdt_object_lock0+0x339/0xaf0 [mdt] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.693993] [] ? mdt_blocking_ast+0x0/0x2a0 [mdt] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.701235] [] ? ldlm_completion_ast+0x0/0x960 [ptlrpc] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.709054] [] mdt_object_lock+0x14/0x20 [mdt] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.715997] [] mdt_object_find_lock+0x61/0x170 [mdt] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.723529] [] mdt_reint_open+0x8cc/0x21c0 [mdt] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.730673] [] ? upcall_cache_get_entry+0x28e/0x860 [libcfs] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.739098] [] ? lustre_msg_add_version+0x6c/0xc0 [ptlrpc] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.747246] [] ? lu_ucred+0x20/0x30 [obdclass] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.754190] [] ? mdt_ucred+0x15/0x20 [mdt] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.760743] [] ? mdt_root_squash+0x2c/0x410 [mdt] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.767979] [] mdt_reint_rec+0x41/0xe0 [mdt] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.774728] [] mdt_reint_internal+0x4c3/0x780 [mdt] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.782157] [] mdt_intent_reint+0x1f0/0x530 [mdt] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.789389] [] mdt_intent_policy+0x39e/0x720 [mdt] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.796735] [] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.804373] [] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.812388] [] mdt_enqueue+0x46/0xe0 [mdt] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.818940] [] mdt_handle_common+0x647/0x16d0 [mdt] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.826390] [] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.834702] [] mds_regular_handle+0x15/0x20 [mdt] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.841955] [] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.850743] [] ? cfs_timer_arm+0xe/0x10 [libcfs] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.857881] [] ? lc_watchdog_touch+0x6f/0x170 [libcfs] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.865621] [] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.873334] [] ? __wake_up+0x53/0x70 Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.879324] [] ptlrpc_main+0xace/0x1700 [ptlrpc] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.886477] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.893603] [] child_rip+0xa/0x20 Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.899300] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.906456] [] ? ptlrpc_main+0x0/0x1700 [ptlrpc] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.913581] [] ? child_rip+0x0/0x20 Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.919451] Jul 17 22:44:33 atlas-mds3.ccs.ornl.gov kernel: [18057.921218] LustreError: dumping log to /tmp/lustre-log.1405651473.15525 Jul 17 22:44:40 atlas-mds3.ccs.ornl.gov kernel: [18064.798619] Lustre: atlas2-MDT0000: Client fdf957bd-4457-0034-ecb5-687b2414db45 (at 31@gni2) reconnecting Jul 17 22:44:40 atlas-mds3.ccs.ornl.gov kernel: [18064.809542] Lustre: Skipped 126 previous similar messages Jul 17 22:48:16 atlas-mds3.ccs.ornl.gov kernel: [18281.091541] Lustre: atlas2-OST0248-osc-MDT0000: slow creates, last=[0x0:0x1:0x0], next=[0x0:0x1:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0, status=0 Jul 17 22:49:52 atlas-mds3.ccs.ornl.gov kernel: [18377.032719] Lustre: atlas2-MDT0000: Client ced5a9bf-ed12-ec09-e41b-c604e72bea77 (at 454@gni3) refused reconnection, still busy with 2 active RPCs Jul 17 22:49:52 atlas-mds3.ccs.ornl.gov kernel: [18377.047536] Lustre: Skipped 107 previous similar messages Jul 17 22:54:51 atlas-mds3.ccs.ornl.gov kernel: [18676.433507] Lustre: atlas2-MDT0000: Client a3c87f05-1b27-9db1-a6f0-8fec93053578 (at 41@gni3) reconnecting Jul 17 22:54:51 atlas-mds3.ccs.ornl.gov kernel: [18676.444438] Lustre: Skipped 108 previous similar messages Jul 17 22:58:16 atlas-mds3.ccs.ornl.gov kernel: [18881.682283] Lustre: atlas2-OST0249-osc-MDT0000: slow creates, last=[0x0:0x1:0x0], next=[0x0:0x1:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0, status=0 Jul 17 22:59:52 atlas-mds3.ccs.ornl.gov kernel: [18977.692978] Lustre: atlas2-MDT0000: Client b34e6fe5-e636-63f4-b861-6f8cbe300c81 (at 10.36.205.217@o2ib) refused reconnection, still busy with 1 active RPCs Jul 17 22:59:52 atlas-mds3.ccs.ornl.gov kernel: [18977.708769] Lustre: Skipped 108 previous similar messages Jul 17 23:04:51 atlas-mds3.ccs.ornl.gov kernel: [19277.010061] Lustre: atlas2-MDT0000: Client a3c87f05-1b27-9db1-a6f0-8fec93053578 (at 41@gni3) reconnecting Jul 17 23:04:51 atlas-mds3.ccs.ornl.gov kernel: [19277.020989] Lustre: Skipped 108 previous similar messages Jul 17 23:08:16 atlas-mds3.ccs.ornl.gov kernel: [19482.275016] Lustre: atlas2-OST024a-osc-MDT0000: slow creates, last=[0x0:0x1:0x0], next=[0x0:0x1:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0, status=0 Jul 17 23:09:53 atlas-mds3.ccs.ornl.gov kernel: [19579.547384] Lustre: atlas2-MDT0000: Client 5fa6cc5c-aa15-ff91-95a8-df9e80e8f453 (at 148@gni3) refused reconnection, still busy with 1 active RPCs Jul 17 23:09:53 atlas-mds3.ccs.ornl.gov kernel: [19579.562197] Lustre: Skipped 110 previous similar messages Jul 17 23:14:51 atlas-mds3.ccs.ornl.gov kernel: [19877.587150] Lustre: atlas2-MDT0000: Client a3c87f05-1b27-9db1-a6f0-8fec93053578 (at 41@gni3) reconnecting Jul 17 23:14:51 atlas-mds3.ccs.ornl.gov kernel: [19877.598074] Lustre: Skipped 107 previous similar messages Jul 17 23:18:16 atlas-mds3.ccs.ornl.gov kernel: [20082.866230] Lustre: atlas2-OST024b-osc-MDT0000: slow creates, last=[0x0:0x1:0x0], next=[0x0:0x1:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0, status=0 Jul 17 23:19:53 atlas-mds3.ccs.ornl.gov kernel: [20180.123134] Lustre: atlas2-MDT0000: Client 5fa6cc5c-aa15-ff91-95a8-df9e80e8f453 (at 148@gni3) refused reconnection, still busy with 1 active RPCs Jul 17 23:19:53 atlas-mds3.ccs.ornl.gov kernel: [20180.137947] Lustre: Skipped 119 previous similar messages Jul 17 23:24:51 atlas-mds3.ccs.ornl.gov kernel: [20478.162482] Lustre: atlas2-MDT0000: Client a3c87f05-1b27-9db1-a6f0-8fec93053578 (at 41@gni3) reconnecting Jul 17 23:24:51 atlas-mds3.ccs.ornl.gov kernel: [20478.173398] Lustre: Skipped 107 previous similar messages Jul 17 23:28:16 atlas-mds3.ccs.ornl.gov kernel: [20683.456518] Lustre: atlas2-OST024c-osc-MDT0000: slow creates, last=[0x0:0x1:0x0], next=[0x0:0x1:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0, status=0 Jul 17 23:28:35 atlas-mds1.ccs.ornl.gov kernel: [818904.380741] LustreError: 35623:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 198283061440, new: 198283061439 replay: 0. see LU-617. Jul 17 23:28:35 atlas-mds1.ccs.ornl.gov kernel: [818904.418658] LustreError: 35623:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 198283063678, new: 198283063677 replay: 0. see LU-617. Jul 17 23:29:33 atlas-mds1.ccs.ornl.gov kernel: [818963.360931] LustreError: 22501:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 198283088941, new: 198283088940 replay: 0. see LU-617. Jul 17 23:29:33 atlas-mds1.ccs.ornl.gov kernel: [818963.378842] LustreError: 22501:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Skipped 8 previous similar messages Jul 17 23:29:34 atlas-mds1.ccs.ornl.gov kernel: [818964.377221] LustreError: 24479:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 198283094967, new: 198283094966 replay: 0. see LU-617. Jul 17 23:29:34 atlas-mds1.ccs.ornl.gov kernel: [818964.395147] LustreError: 24479:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Skipped 1 previous similar message Jul 17 23:29:53 atlas-mds3.ccs.ornl.gov kernel: [20780.707465] Lustre: atlas2-MDT0000: Client 5fa6cc5c-aa15-ff91-95a8-df9e80e8f453 (at 148@gni3) refused reconnection, still busy with 1 active RPCs Jul 17 23:29:53 atlas-mds3.ccs.ornl.gov kernel: [20780.722283] Lustre: Skipped 128 previous similar messages Jul 17 23:31:16 atlas-mds1.ccs.ornl.gov kernel: [819065.645958] LustreError: 12025:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 198283130976, new: 198283130975 replay: 0. see LU-617. Jul 17 23:31:16 atlas-mds1.ccs.ornl.gov kernel: [819065.663891] LustreError: 12025:0:(mdt_recovery.c:418:mdt_last_rcvd_update()) Skipped 13 previous similar messages Jul 17 23:31:24 atlas-mds3.ccs.ornl.gov kernel: [20870.881816] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.36.205.207@o2ib ns: mdt-atlas2-MDT0000_UUID lock: ffff883f42ff8480/0xf35a0587ba12a9ea lrc: 3/0,0 mode: PR/PR res: [0x2000d5462:0xc315:0x0].0 bits 0x13 rrc: 3 type: IBT flags: 0x20 nid: 10.36.205.207@o2ib remote: 0xd4524341fde12fef expref: 2000 pid: 16168 timeout: 4315516687 lvb_type: 0 used 0 Jul 17 23:34:52 atlas-mds3.ccs.ornl.gov kernel: [21079.557499] Lustre: atlas2-MDT0000: Client f751480e-482d-4566-5dc1-5bb84d5481fd (at 10.36.247.134@o2ib) reconnecting Jul 17 23:34:52 atlas-mds3.ccs.ornl.gov kernel: [21079.569487] Lustre: Skipped 128 previous similar messages Jul 17 23:38:16 atlas-mds3.ccs.ornl.gov kernel: [21284.047340] Lustre: atlas2-OST024d-osc-MDT0000: slow creates, last=[0x0:0x1:0x0], next=[0x0:0x1:0x0], reserved=0, syn_changes=0, syn_rpc_in_progress=0, status=0 Jul 17 23:39:57 atlas-mds3.ccs.ornl.gov kernel: [21384.515672] Lustre: atlas2-MDT0000: Client 0fec8137-2e19-aee1-6e02-696fad79258f (at 142@gni3) refused reconnection, still busy with 1 active RPCs Jul 17 23:39:57 atlas-mds3.ccs.ornl.gov kernel: [21384.530514] Lustre: Skipped 106 previous similar messages Jul 17 23:41:04 atlas-mds3.ccs.ornl.gov kernel: [21451.437452] LustreError: 0:0:(ldlm_lockd.c:402:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10.36.225.5@o2ib ns: mdt-atlas2-MDT0000_UUID lock: ffff881dc22416c0/0xf35a0587bbc8e66c lrc: 3/0,0 mode: PR/PR res: [0x200000400:0xa3:0x0].0 bits 0x13 rrc: 10 type: IBT flags: 0x200000000020 nid: 10.36.225.5@o2ib remote: 0xf30cd072d76075ce expref: 541 pid: 15676 timeout: 4316096832 lvb_type: 0 used 0 Jul 17 23:41:11 atlas-oss4a5.ccs.ornl.gov kernel: [1939654.584760] LustreError: 32793:0:(ldlm_resource.c:1165:ldlm_resource_get()) atlas2-OST031c: lvbo_init failed for resource 0x1ea9af:0x0: rc = -2 Jul 17 23:41:11 atlas-oss3h4.ccs.ornl.gov kernel: [1939639.166076] LustreError: 99752:0:(ldlm_resource.c:1165:ldlm_resource_get()) atlas2-OST015b: lvbo_init failed for resource 0x20d9e1:0x0: rc = -2 Jul 17 23:41:11 atlas-oss3e1.ccs.ornl.gov kernel: [1939638.128236] LustreError: 72868:0:(ldlm_resource.c:1165:ldlm_resource_get()) atlas2-OST01d0: lvbo_init failed for resource 0x2013de:0x0: rc = -2 Jul 17 23:41:11 atlas-oss4i5.ccs.ornl.gov kernel: [1939643.377607] LustreError: 40073:0:(ldlm_resource.c:1165:ldlm_resource_get()) atlas2-OST035c: lvbo_init failed for resource 0x1ed23b:0x0: rc = -2