Jul 16 12:33:22 iws29 LDAPOTP-AUTH[8057]: root@iws1.llnl.gov as root: cmd='mkdir -p /p/iws28_1; mount -t lustre /dev/mapper/iws28_1 /p/iws28_1'
Jul 16 12:33:22 iws29 kernel: LDISKFS-fs warning (device dm-2): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jul 16 12:33:22 iws29 kernel: 
Jul 16 12:34:01 iws29 kernel: LustreError: 137-5: lustre-OST000a_UUID: not available for connect from 192.168.124.207@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jul 16 12:34:01 iws29 kernel: LustreError: Skipped 2 previous similar messages
Jul 16 12:34:01 iws29 kernel: LustreError: 137-5: lustre-OST000a_UUID: not available for connect from 192.168.124.126@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jul 16 12:34:01 iws29 kernel: LustreError: Skipped 5 previous similar messages
Jul 16 12:34:02 iws29 kernel: LustreError: 137-5: lustre-OST000a_UUID: not available for connect from 192.168.124.217@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jul 16 12:34:02 iws29 kernel: LustreError: Skipped 22 previous similar messages
Jul 16 12:34:07 iws29 kernel: LDISKFS-fs (dm-2): recovery complete
Jul 16 12:34:07 iws29 kernel: LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. quota=on. Opts: 
Jul 16 12:34:07 iws29 kernel: Lustre: lustre-OST000a: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450
Jul 16 12:34:07 iws29 LDAPOTP-AUTH[8120]: root@iws1.llnl.gov as root: cmd='mkdir -p /p/iws28_2; mount -t lustre /dev/mapper/iws28_2 /p/iws28_2'
Jul 16 12:34:07 iws29 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jul 16 12:34:07 iws29 kernel: 
Jul 16 12:34:09 iws29 kernel: Lustre: lustre-OST000a: Will be in recovery for at least 2:30, or until 258 clients reconnect
Jul 16 12:34:09 iws29 kernel: LustreError: 137-5: lustre-OST0016_UUID: not available for connect from 192.168.124.151@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jul 16 12:34:09 iws29 kernel: LustreError: Skipped 6 previous similar messages
Jul 16 12:34:13 iws29 kernel: LustreError: 137-5: lustre-OST0016_UUID: not available for connect from 192.168.124.86@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jul 16 12:34:13 iws29 kernel: LustreError: Skipped 38 previous similar messages
Jul 16 12:34:22 iws29 kernel: LustreError: 137-5: lustre-OST0016_UUID: not available for connect from 192.168.124.119@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jul 16 12:34:22 iws29 kernel: LustreError: Skipped 83 previous similar messages
Jul 16 12:34:38 iws29 kernel: LustreError: 137-5: lustre-OST0016_UUID: not available for connect from 192.168.124.8@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jul 16 12:34:38 iws29 kernel: LustreError: Skipped 66 previous similar messages
Jul 16 12:34:56 iws29 kernel: LDISKFS-fs (dm-1): recovery complete
Jul 16 12:34:56 iws29 kernel: LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. quota=on. Opts: 
Jul 16 12:34:56 iws29 kernel: Lustre: lustre-OST0016: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450
Jul 16 12:34:56 iws29 LDAPOTP-AUTH[8183]: root@iws1.llnl.gov as root: cmd='mkdir -p /p/iws28_3; mount -t lustre /dev/mapper/iws28_3 /p/iws28_3'
Jul 16 12:34:57 iws29 kernel: LDISKFS-fs warning (device dm-5): ldiskfs_multi_mount_protect: MMP interval 42 higher than expected, please wait.
Jul 16 12:34:57 iws29 kernel: 
Jul 16 12:34:59 iws29 kernel: Lustre: lustre-OST0016: Will be in recovery for at least 2:30, or until 258 clients reconnect
Jul 16 12:35:01 iws29 kernel: program snmpd is using a deprecated SCSI ioctl, please convert it to SG_IO
Jul 16 12:35:05 iws29 kernel: Lustre: ost: This server is not able to keep up with request traffic (cpu-bound).
Jul 16 12:35:05 iws29 kernel: Lustre: 7903:0:(service.c:1511:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=0, svcEst=45, delay=0(jiff)
Jul 16 12:35:05 iws29 kernel: Lustre: 7903:0:(service.c:1308:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-11s), not sending early reply. Consider increasing at_early_margin (5)?  req@ffff8805324da800 x1473716773594080/t0(0) o400->b1211cf5-fced-b5d5-1f8a-9d542b4cf0ee@192.168.124.146@o2ib:0/0 lens 224/0 e 1 to 0 dl 1405539294 ref 2 fl Complete:H/c0/ffffffff rc 0/-1
Jul 16 12:35:06 iws29 kernel: Lustre: ost: This server is not able to keep up with request traffic (cpu-bound).
Jul 16 12:35:06 iws29 kernel: Lustre: Skipped 2 previous similar messages
Jul 16 12:35:06 iws29 kernel: Lustre: 7903:0:(service.c:1511:ptlrpc_at_check_timed()) earlyQ=2 reqQ=0 recA=0, svcEst=45, delay=0(jiff)
Jul 16 12:35:06 iws29 kernel: Lustre: 7903:0:(service.c:1511:ptlrpc_at_check_timed()) Skipped 2 previous similar messages
Jul 16 12:35:06 iws29 kernel: Lustre: 7903:0:(service.c:1308:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-11s), not sending early reply. Consider increasing at_early_margin (5)?  req@ffff880603d6fc00 x1473717382844492/t0(0) o400->a4c991e3-2a86-3c26-b7af-348fd1f57fb6@192.168.124.151@o2ib:0/0 lens 224/0 e 1 to 0 dl 1405539295 ref 2 fl Complete:H/c0/ffffffff rc 0/-1
Jul 16 12:35:06 iws29 kernel: Lustre: 7903:0:(service.c:1308:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages
Jul 16 12:35:07 iws29 kernel: Lustre: ost: This server is not able to keep up with request traffic (cpu-bound).
Jul 16 12:35:07 iws29 kernel: Lustre: Skipped 4 previous similar messages
Jul 16 12:35:07 iws29 kernel: Lustre: 7543:0:(service.c:1511:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=0, svcEst=45, delay=0(jiff)
Jul 16 12:35:07 iws29 kernel: Lustre: 7543:0:(service.c:1511:ptlrpc_at_check_timed()) Skipped 4 previous similar messages
Jul 16 12:35:07 iws29 kernel: Lustre: 7543:0:(service.c:1308:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-11s), not sending early reply. Consider increasing at_early_margin (5)?  req@ffff880d90123400 x1473717097877236/t0(0) o400->d5807a7b-6003-0a1e-7718-5646af09c44d@192.168.124.113@o2ib:0/0 lens 224/0 e 1 to 0 dl 1405539296 ref 2 fl Complete:H/c0/ffffffff rc 0/-1
Jul 16 12:35:07 iws29 kernel: Lustre: 7543:0:(service.c:1308:ptlrpc_at_send_early_reply()) Skipped 11 previous similar messages
Jul 16 12:35:09 iws29 kernel: Lustre: ost: This server is not able to keep up with request traffic (cpu-bound).
Jul 16 12:35:09 iws29 kernel: Lustre: Skipped 7 previous similar messages
Jul 16 12:35:09 iws29 kernel: Lustre: 7534:0:(service.c:1511:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=0, svcEst=45, delay=0(jiff)
Jul 16 12:35:09 iws29 kernel: Lustre: 7593:0:(service.c:1308:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-11s), not sending early reply. Consider increasing at_early_margin (5)?  req@ffff880d90741400 x1473717064223520/t0(0) o400->396015fe-375c-4be4-5d40-ff0a5b819d26@192.168.124.148@o2ib:0/0 lens 224/0 e 1 to 0 dl 1405539298 ref 2 fl Complete:H/c0/ffffffff rc 0/-1
Jul 16 12:35:09 iws29 kernel: Lustre: 7593:0:(service.c:1308:ptlrpc_at_send_early_reply()) Skipped 14 previous similar messages
Jul 16 12:35:09 iws29 kernel: Lustre: 7534:0:(service.c:1511:ptlrpc_at_check_timed()) Skipped 8 previous similar messages
Jul 16 12:35:12 iws29 kernel: LustreError: 137-5: lustre-OST0022_UUID: not available for connect from 192.168.124.119@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jul 16 12:35:12 iws29 kernel: LustreError: Skipped 366 previous similar messages
Jul 16 12:35:13 iws29 kernel: Lustre: ost: This server is not able to keep up with request traffic (cpu-bound).
Jul 16 12:35:13 iws29 kernel: Lustre: Skipped 22 previous similar messages
Jul 16 12:35:13 iws29 kernel: Lustre: 7540:0:(service.c:1511:ptlrpc_at_check_timed()) earlyQ=3 reqQ=0 recA=0, svcEst=45, delay=0(jiff)
Jul 16 12:35:13 iws29 kernel: Lustre: 7540:0:(service.c:1511:ptlrpc_at_check_timed()) Skipped 21 previous similar messages
Jul 16 12:35:13 iws29 kernel: Lustre: 7540:0:(service.c:1308:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-11s), not sending early reply. Consider increasing at_early_margin (5)?  req@ffff880d9683a000 x1473717066677324/t0(0) o400->80f2f5e5-aff7-cc60-698d-ef586be9948f@192.168.124.114@o2ib:0/0 lens 224/0 e 1 to 0 dl 1405539302 ref 2 fl Complete:H/c0/ffffffff rc 0/-1
Jul 16 12:35:13 iws29 kernel: Lustre: 7540:0:(service.c:1308:ptlrpc_at_send_early_reply()) Skipped 45 previous similar messages
Jul 16 12:35:16 iws29 kernel: LustreError: 8118:0:(tgt_handler.c:1980:tgt_brw_write()) lustre-OST000a: Dropping timed-out write from 12345-192.168.124.153@o2ib because locking object 0x0:11302148 took 61 seconds (limit was 45).
Jul 16 12:35:16 iws29 kernel: Lustre: lustre-OST000a: Bulk IO write error with 31e19eaa-f613-e151-fdca-c418a7d93c81 (at 192.168.124.153@o2ib), client will retry: rc -110
Jul 16 12:35:21 iws29 kernel: Lustre: ost_io: This server is not able to keep up with request traffic (cpu-bound).
Jul 16 12:35:21 iws29 kernel: Lustre: Skipped 1 previous similar message
Jul 16 12:35:21 iws29 kernel: Lustre: 7771:0:(service.c:1511:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=0, svcEst=45, delay=0(jiff)
Jul 16 12:35:21 iws29 kernel: Lustre: 7771:0:(service.c:1511:ptlrpc_at_check_timed()) Skipped 1 previous similar message
Jul 16 12:35:21 iws29 kernel: Lustre: 7771:0:(service.c:1308:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-11s), not sending early reply. Consider increasing at_early_margin (5)?  req@ffff880533504000 x1473702935594124/t0(4320571190) o4->c43f355b-28da-58ec-e885-107b4a81a73d@192.168.124.78@o2ib:0/0 lens 488/0 e 1 to 0 dl 1405539310 ref 2 fl Complete:/4/ffffffff rc 0/-1
Jul 16 12:35:21 iws29 kernel: Lustre: 7771:0:(service.c:1308:ptlrpc_at_send_early_reply()) Skipped 3 previous similar messages
Jul 16 12:35:38 iws29 kernel: Lustre: ost: This server is not able to keep up with request traffic (cpu-bound).
Jul 16 12:35:38 iws29 kernel: Lustre: Skipped 53 previous similar messages
Jul 16 12:35:38 iws29 kernel: Lustre: 7894:0:(service.c:1511:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=0, svcEst=45, delay=0(jiff)
Jul 16 12:35:38 iws29 kernel: Lustre: 7894:0:(service.c:1511:ptlrpc_at_check_timed()) Skipped 53 previous similar messages
Jul 16 12:35:38 iws29 kernel: Lustre: 7894:0:(service.c:1308:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-11s), not sending early reply. Consider increasing at_early_margin (5)?  req@ffff88053432d000 x1473702654146796/t0(0) o400->ef3ab032-9217-85f9-3b65-253a182d2f1b@192.168.124.57@o2ib:0/0 lens 224/0 e 1 to 0 dl 1405539327 ref 2 fl Complete:H/c0/ffffffff rc 0/-1
Jul 16 12:35:38 iws29 kernel: Lustre: 7894:0:(service.c:1308:ptlrpc_at_send_early_reply()) Skipped 133 previous similar messages
Jul 16 12:35:40 iws29 kernel: LDISKFS-fs (dm-5): recovery complete
Jul 16 12:35:40 iws29 kernel: LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. quota=on. Opts: 
Jul 16 12:35:41 iws29 kernel: Lustre: lustre-OST0022: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450
Jul 16 12:35:41 iws29 kernel: Lustre: lustre-OST0022: Will be in recovery for at least 2:30, or until 258 clients reconnect
Jul 16 12:35:41 iws29 LDAPOTP-AUTH[8248]: root@iws1.llnl.gov as root: cmd='lctl dl'
Jul 16 12:35:55 iws29 kernel: Lustre: lustre-OST000a: Client 0edce5a1-3069-3feb-5de3-4208c9838f8a (at 192.168.124.118@o2ib) reconnecting, waiting for 258 clients in recovery for 1:52
Jul 16 12:35:55 iws29 kernel: Lustre: lustre-OST000a: Client b1211cf5-fced-b5d5-1f8a-9d542b4cf0ee (at 192.168.124.146@o2ib) reconnecting, waiting for 258 clients in recovery for 1:51
Jul 16 12:35:55 iws29 kernel: Lustre: Skipped 2 previous similar messages
Jul 16 12:35:56 iws29 kernel: LustreError: 7556:0:(ldlm_lib.c:2302:target_queue_recovery_request()) @@@ dropping resent queued req  req@ffff88052fb9d400 x1473717364235580/t0(4320571308) o4->5caf10ee-c3ea-8f73-8c45-4a7b1f3740d8@192.168.124.96@o2ib:0/0 lens 504/0 e 0 to 0 dl 1405539417 ref 2 fl Interpret:/6/ffffffff rc 0/-1
Jul 16 12:35:57 iws29 kernel: Lustre: lustre-OST000a: Client 3ea6a679-93fd-4fb6-a948-24f72f59ad7f (at 192.168.124.101@o2ib) reconnecting, waiting for 258 clients in recovery for 1:50
Jul 16 12:35:57 iws29 kernel: Lustre: Skipped 10 previous similar messages
Jul 16 12:35:59 iws29 kernel: Lustre: lustre-OST000a: Client 692bc62a-0573-b07b-dd27-6e9a34973bb0 (at 192.168.124.86@o2ib) reconnecting, waiting for 258 clients in recovery for 1:48
Jul 16 12:35:59 iws29 kernel: Lustre: Skipped 13 previous similar messages
Jul 16 12:36:00 iws29 kernel: LustreError: 7780:0:(ldlm_lib.c:2302:target_queue_recovery_request()) @@@ dropping resent queued req  req@ffff880d8c8f8000 x1473717201322780/t0(4320571202) o4->40311b72-6fbb-58bb-0ed2-a345ee582320@192.168.124.181@o2ib:0/0 lens 488/0 e 0 to 0 dl 1405539421 ref 2 fl Interpret:/6/ffffffff rc 0/-1
Jul 16 12:36:02 iws29 kernel: LustreError: 8118:0:(tgt_handler.c:1980:tgt_brw_write()) lustre-OST000a: Dropping timed-out write from 12345-192.168.123.248@o2ib because locking object 0x0:11292446 took 95 seconds (limit was 45).
Jul 16 12:36:02 iws29 kernel: LustreError: 8118:0:(tgt_handler.c:1980:tgt_brw_write()) Skipped 1 previous similar message
Jul 16 12:36:02 iws29 kernel: Lustre: lustre-OST000a: Bulk IO write error with 289842ab-19e9-1ed0-06ea-4090dace0d92 (at 192.168.123.248@o2ib), client will retry: rc -110
Jul 16 12:36:02 iws29 kernel: Lustre: Skipped 1 previous similar message
Jul 16 12:36:03 iws29 kernel: Lustre: lustre-OST000a: Client d460b957-b002-89b2-5439-bb40897da0e4 (at 192.168.124.150@o2ib) reconnecting, waiting for 258 clients in recovery for 2:29
Jul 16 12:36:03 iws29 kernel: Lustre: Skipped 46 previous similar messages
Jul 16 12:36:11 iws29 kernel: Lustre: ost: This server is not able to keep up with request traffic (cpu-bound).
Jul 16 12:36:11 iws29 kernel: Lustre: Skipped 103 previous similar messages
Jul 16 12:36:11 iws29 kernel: Lustre: 7894:0:(service.c:1511:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=0, svcEst=45, delay=0(jiff)
Jul 16 12:36:11 iws29 kernel: Lustre: 7894:0:(service.c:1511:ptlrpc_at_check_timed()) Skipped 103 previous similar messages
Jul 16 12:36:11 iws29 kernel: Lustre: 7894:0:(service.c:1308:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-11s), not sending early reply. Consider increasing at_early_margin (5)?  req@ffff88050c340c00 x1473702936016252/t0(0) o400->c43f355b-28da-58ec-e885-107b4a81a73d@192.168.124.78@o2ib:0/0 lens 224/0 e 1 to 0 dl 1405539360 ref 2 fl Complete:H/c0/ffffffff rc 0/-1
Jul 16 12:36:11 iws29 kernel: Lustre: 7894:0:(service.c:1308:ptlrpc_at_send_early_reply()) Skipped 222 previous similar messages
Jul 16 12:36:11 iws29 kernel: Lustre: lustre-OST000a: Client 359d47b9-bc93-e739-ad26-6bb99d472796 (at 192.168.119.13@o2ib) reconnecting, waiting for 258 clients in recovery for 2:21
Jul 16 12:36:11 iws29 kernel: Lustre: Skipped 4 previous similar messages
Jul 16 12:36:11 iws29 kernel: LustreError: 8118:0:(tgt_handler.c:1980:tgt_brw_write()) lustre-OST000a: Dropping timed-out write from 12345-192.168.119.16@o2ib because locking object 0x0:11302154 took 104 seconds (limit was 45).
Jul 16 12:36:11 iws29 kernel: LustreError: 8118:0:(tgt_handler.c:1980:tgt_brw_write()) Skipped 13 previous similar messages
Jul 16 12:36:11 iws29 kernel: Lustre: lustre-OST000a: Bulk IO write error with f0399adc-982c-54eb-663e-bd2d9c823d55 (at 192.168.119.16@o2ib), client will retry: rc -110
Jul 16 12:36:11 iws29 kernel: Lustre: Skipped 13 previous similar messages
Jul 16 12:36:15 iws29 kernel: LustreError: 8181:0:(tgt_handler.c:1980:tgt_brw_write()) lustre-OST0016: Dropping timed-out write from 12345-192.168.124.72@o2ib because locking object 0x0:11273430 took 59 seconds (limit was 45).
Jul 16 12:36:15 iws29 kernel: LustreError: 8181:0:(tgt_handler.c:1980:tgt_brw_write()) Skipped 1 previous similar message
Jul 16 12:36:15 iws29 kernel: Lustre: lustre-OST0016: Bulk IO write error with d5b2a258-bd76-13b3-cc4c-174bd274aa7d (at 192.168.124.72@o2ib), client will retry: rc -110
Jul 16 12:36:15 iws29 kernel: Lustre: Skipped 1 previous similar message
Jul 16 12:36:27 iws29 kernel: Lustre: lustre-OST000a: Client ccae8372-fa2e-cc9e-2d07-1312ac60efa5 (at 192.168.124.243@o2ib) reconnecting, waiting for 258 clients in recovery for 2:15
Jul 16 12:36:27 iws29 kernel: Lustre: Skipped 131 previous similar messages
Jul 16 12:36:37 iws29 kernel: Lustre: lustre-OST0022: deleting orphan objects from 0x0:11302007 to 0x0:11303297
Jul 16 12:36:37 iws29 kernel: Lustre: lustre-OST0022: Recovery over after 0:56, of 258 clients 258 recovered and 0 were evicted.
Jul 16 12:36:50 iws29 kernel: LustreError: 8045:0:(ldlm_lib.c:2302:target_queue_recovery_request()) @@@ dropping resent queued req  req@ffff88050ec1ac00 x1473716625961368/t0(4320363838) o4->0edce5a1-3069-3feb-5de3-4208c9838f8a@192.168.124.118@o2ib:0/0 lens 488/0 e 0 to 0 dl 1405539471 ref 2 fl Interpret:/6/ffffffff rc 0/-1
Jul 16 12:36:53 iws29 kernel: LustreError: 7771:0:(ldlm_lib.c:2302:target_queue_recovery_request()) @@@ dropping resent queued req  req@ffff8805344c2800 x1473717649558184/t0(4320363832) o4->28b574f2-b916-9f45-5bf2-5fa229a18e41@192.168.124.201@o2ib:0/0 lens 488/0 e 0 to 0 dl 1405539474 ref 2 fl Interpret:/6/ffffffff rc 0/-1
Jul 16 12:36:53 iws29 kernel: LustreError: 7771:0:(ldlm_lib.c:2302:target_queue_recovery_request()) Skipped 2 previous similar messages
Jul 16 12:37:01 iws29 kernel: Lustre: lustre-OST0016: Client c43f355b-28da-58ec-e885-107b4a81a73d (at 192.168.124.78@o2ib) reconnecting, waiting for 258 clients in recovery for 1:54
Jul 16 12:37:01 iws29 kernel: Lustre: Skipped 223 previous similar messages
Jul 16 12:37:07 iws29 kernel: LustreError: 8181:0:(tgt_handler.c:1980:tgt_brw_write()) lustre-OST0016: Dropping timed-out write from 12345-192.168.124.114@o2ib because locking object 0x0:11276654 took 122 seconds (limit was 45).
Jul 16 12:37:07 iws29 kernel: LustreError: 8181:0:(tgt_handler.c:1980:tgt_brw_write()) Skipped 5 previous similar messages
Jul 16 12:37:07 iws29 kernel: Lustre: lustre-OST0016: Bulk IO write error with 80f2f5e5-aff7-cc60-698d-ef586be9948f (at 192.168.124.114@o2ib), client will retry: rc -110
Jul 16 12:37:07 iws29 kernel: Lustre: Skipped 5 previous similar messages
Jul 16 12:37:18 iws29 kernel: Lustre: ost: This server is not able to keep up with request traffic (cpu-bound).
Jul 16 12:37:18 iws29 kernel: Lustre: Skipped 102 previous similar messages
Jul 16 12:37:18 iws29 kernel: Lustre: 7593:0:(service.c:1511:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=0, svcEst=45, delay=0(jiff)
Jul 16 12:37:18 iws29 kernel: Lustre: 7593:0:(service.c:1511:ptlrpc_at_check_timed()) Skipped 102 previous similar messages
Jul 16 12:37:18 iws29 kernel: Lustre: 7593:0:(service.c:1308:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-11s), not sending early reply. Consider increasing at_early_margin (5)?  req@ffff880d7f824000 x1473702851602796/t0(0) o400->7bdef850-6afc-b4dd-6d8d-63ee1496c44d@192.168.124.67@o2ib:0/0 lens 224/0 e 1 to 0 dl 1405539427 ref 2 fl Complete:H/c0/ffffffff rc 0/-1
Jul 16 12:37:18 iws29 kernel: Lustre: 7593:0:(service.c:1308:ptlrpc_at_send_early_reply()) Skipped 175 previous similar messages
Jul 16 12:37:32 iws29 kernel: LustreError: 7697:0:(ldlm_lib.c:2302:target_queue_recovery_request()) @@@ dropping resent queued req  req@ffff880d514e2400 x1473702059672624/t0(4320363839) o4->e41b94a0-41d9-eb7d-1fc3-0726af6dd09f@192.168.118.129@o2ib:0/0 lens 488/0 e 0 to 0 dl 1405539513 ref 2 fl Interpret:/6/ffffffff rc 0/-1
Jul 16 12:37:32 iws29 kernel: LustreError: 7697:0:(ldlm_lib.c:2302:target_queue_recovery_request()) Skipped 7 previous similar messages
Jul 16 12:37:42 iws29 kernel: LustreError: 7771:0:(ldlm_lib.c:2302:target_queue_recovery_request()) @@@ dropping resent queued req  req@ffff8804c842b400 x1473717364235580/t0(4320571308) o4->5caf10ee-c3ea-8f73-8c45-4a7b1f3740d8@192.168.124.96@o2ib:0/0 lens 504/0 e 0 to 0 dl 1405539523 ref 2 fl Interpret:/6/ffffffff rc 0/-1
Jul 16 12:37:42 iws29 kernel: LustreError: 7771:0:(ldlm_lib.c:2302:target_queue_recovery_request()) Skipped 2 previous similar messages
Jul 16 12:37:46 iws29 kernel: LustreError: 8118:0:(tgt_handler.c:1980:tgt_brw_write()) lustre-OST000a: Dropping timed-out write from 12345-192.168.124.78@o2ib because locking object 0x0:11272550 took 77 seconds (limit was 45).
Jul 16 12:37:46 iws29 kernel: LustreError: 8118:0:(tgt_handler.c:1980:tgt_brw_write()) Skipped 4 previous similar messages
Jul 16 12:37:46 iws29 kernel: Lustre: lustre-OST000a: Bulk IO write error with c43f355b-28da-58ec-e885-107b4a81a73d (at 192.168.124.78@o2ib), client will retry: rc -110
Jul 16 12:37:46 iws29 kernel: Lustre: Skipped 4 previous similar messages
Jul 16 12:37:59 iws29 kernel: LustreError: 7562:0:(ldlm_lib.c:2302:target_queue_recovery_request()) @@@ dropping resent queued req  req@ffff880d4f0d9800 x1473702851135940/t0(4320571240) o4->a8b955a7-2fa6-1c8e-7a28-e4b17881664f@192.168.124.179@o2ib:0/0 lens 488/0 e 0 to 0 dl 1405539540 ref 2 fl Interpret:/6/ffffffff rc 0/-1
Jul 16 12:37:59 iws29 kernel: LustreError: 7562:0:(ldlm_lib.c:2302:target_queue_recovery_request()) Skipped 2 previous similar messages
Jul 16 12:38:06 iws29 kernel: Lustre: lustre-OST0016: Client 2198ec6f-2360-9b89-bca4-5f8cbc095ab1 (at 192.168.124.189@o2ib) reconnecting, waiting for 258 clients in recovery for 1:41
Jul 16 12:38:06 iws29 kernel: Lustre: Skipped 176 previous similar messages
Jul 16 12:38:15 iws29 kernel: LustreError: 8118:0:(tgt_handler.c:1980:tgt_brw_write()) lustre-OST000a: Dropping timed-out write from 12345-192.168.124.72@o2ib because locking object 0x0:11272553 took 106 seconds (limit was 45).
Jul 16 12:38:15 iws29 kernel: LustreError: 8118:0:(tgt_handler.c:1980:tgt_brw_write()) Skipped 6 previous similar messages
Jul 16 12:38:15 iws29 kernel: Lustre: lustre-OST000a: Bulk IO write error with d5b2a258-bd76-13b3-cc4c-174bd274aa7d (at 192.168.124.72@o2ib), client will retry: rc -110
Jul 16 12:38:15 iws29 kernel: Lustre: Skipped 6 previous similar messages
Jul 16 12:38:38 iws29 kernel: LustreError: 8034:0:(ldlm_lib.c:2302:target_queue_recovery_request()) @@@ dropping resent queued req  req@ffff8804c779cc00 x1473702102125048/t0(4320363842) o4->463c2bac-1c2a-8f97-9623-b92d3fdd9f59@192.168.118.112@o2ib:0/0 lens 488/0 e 0 to 0 dl 1405539579 ref 2 fl Interpret:/6/ffffffff rc 0/-1
Jul 16 12:38:38 iws29 kernel: LustreError: 8034:0:(ldlm_lib.c:2302:target_queue_recovery_request()) Skipped 4 previous similar messages
Jul 16 12:38:47 iws29 kernel: LustreError: 8181:0:(tgt_handler.c:1980:tgt_brw_write()) lustre-OST0016: Dropping timed-out write from 12345-192.168.124.201@o2ib because locking object 0x0:11303023 took 225 seconds (limit was 45).
Jul 16 12:38:47 iws29 kernel: LustreError: 8181:0:(tgt_handler.c:1980:tgt_brw_write()) Skipped 10 previous similar messages
Jul 16 12:38:47 iws29 kernel: Lustre: lustre-OST0016: Bulk IO write error with 28b574f2-b916-9f45-5bf2-5fa229a18e41 (at 192.168.124.201@o2ib), client will retry: rc -110
Jul 16 12:38:47 iws29 kernel: Lustre: Skipped 10 previous similar messages
Jul 16 12:39:29 iws29 kernel: Lustre: ost: This server is not able to keep up with request traffic (cpu-bound).
Jul 16 12:39:29 iws29 kernel: Lustre: Skipped 271 previous similar messages
Jul 16 12:39:29 iws29 kernel: Lustre: 7645:0:(service.c:1511:ptlrpc_at_check_timed()) earlyQ=2 reqQ=0 recA=0, svcEst=45, delay=0(jiff)
Jul 16 12:39:29 iws29 kernel: Lustre: 7645:0:(service.c:1511:ptlrpc_at_check_timed()) Skipped 271 previous similar messages
Jul 16 12:39:29 iws29 kernel: Lustre: 7645:0:(service.c:1308:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-11s), not sending early reply. Consider increasing at_early_margin (5)?  req@ffff8804c8375c00 x1473702708725600/t0(0) o400->c07cad3a-e79f-caea-5396-eb373258abac@192.168.123.242@o2ib:0/0 lens 224/0 e 1 to 0 dl 1405539558 ref 2 fl Complete:H/c0/ffffffff rc 0/-1
Jul 16 12:39:29 iws29 kernel: Lustre: 7645:0:(service.c:1308:ptlrpc_at_send_early_reply()) Skipped 646 previous similar messages
Jul 16 12:39:44 iws29 kernel: LustreError: 8034:0:(ldlm_lib.c:2302:target_queue_recovery_request()) @@@ dropping resent queued req  req@ffff8804edd26400 x1473701990309436/t0(4320571271) o4->d77917e0-de49-cace-a792-300bcef262d7@192.168.118.143@o2ib:0/0 lens 488/0 e 0 to 0 dl 1405539645 ref 2 fl Interpret:/6/ffffffff rc 0/-1
Jul 16 12:39:44 iws29 kernel: LustreError: 8034:0:(ldlm_lib.c:2302:target_queue_recovery_request()) Skipped 11 previous similar messages
Jul 16 12:40:01 iws29 kernel: program snmpd is using a deprecated SCSI ioctl, please convert it to SG_IO
Jul 16 12:40:02 iws29 kernel: LustreError: 8118:0:(tgt_handler.c:1980:tgt_brw_write()) lustre-OST000a: Dropping timed-out write from 12345-192.168.124.183@o2ib because locking object 0x0:11292447 took 106 seconds (limit was 45).
Jul 16 12:40:02 iws29 kernel: LustreError: 8118:0:(tgt_handler.c:1980:tgt_brw_write()) Skipped 26 previous similar messages
Jul 16 12:40:02 iws29 kernel: Lustre: lustre-OST000a: Bulk IO write error with ec8bbc42-9e01-af81-a96a-6fb57e969816 (at 192.168.124.183@o2ib), client will retry: rc -110
Jul 16 12:40:02 iws29 kernel: Lustre: Skipped 26 previous similar messages
Jul 16 12:40:19 iws29 kernel: Lustre: lustre-OST0016: Client f8c1a822-1e6e-0610-0d5c-258b967227ea (at 192.168.124.136@o2ib) reconnecting, waiting for 258 clients in recovery for 1:20
Jul 16 12:40:19 iws29 kernel: Lustre: Skipped 670 previous similar messages
Jul 16 12:42:25 iws29 kernel: LustreError: 8045:0:(ldlm_lib.c:2302:target_queue_recovery_request()) @@@ dropping resent queued req  req@ffff88050972f000 x1473717451401480/t0(4320364254) o4->18f03277-453f-a1d8-083a-08f4834b656e@192.168.124.94@o2ib:0/0 lens 504/0 e 0 to 0 dl 1405539805 ref 2 fl Interpret:/6/ffffffff rc 0/-1
Jul 16 12:42:25 iws29 kernel: LustreError: 8045:0:(ldlm_lib.c:2302:target_queue_recovery_request()) Skipped 20 previous similar messages
Jul 16 12:42:30 iws29 kernel: LustreError: 8181:0:(tgt_handler.c:1980:tgt_brw_write()) lustre-OST0016: Dropping timed-out write from 12345-192.168.124.184@o2ib because locking object 0x0:11293322 took 175 seconds (limit was 45).
Jul 16 12:42:30 iws29 kernel: LustreError: 8181:0:(tgt_handler.c:1980:tgt_brw_write()) Skipped 28 previous similar messages
Jul 16 12:42:30 iws29 kernel: Lustre: lustre-OST0016: Bulk IO write error with 0647eae6-48e3-fcc2-379b-af62a00efaf5 (at 192.168.124.184@o2ib), client will retry: rc -110
Jul 16 12:42:30 iws29 kernel: Lustre: Skipped 28 previous similar messages
Jul 16 12:43:55 iws29 kernel: Lustre: ost: This server is not able to keep up with request traffic (cpu-bound).
Jul 16 12:43:55 iws29 kernel: Lustre: Skipped 521 previous similar messages
Jul 16 12:43:55 iws29 kernel: Lustre: 7598:0:(service.c:1511:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=0, svcEst=45, delay=0(jiff)
Jul 16 12:43:55 iws29 kernel: Lustre: 7598:0:(service.c:1511:ptlrpc_at_check_timed()) Skipped 521 previous similar messages
Jul 16 12:43:55 iws29 kernel: Lustre: 7598:0:(service.c:1308:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-11s), not sending early reply. Consider increasing at_early_margin (5)?  req@ffff88052ff36850 x1473716773597204/t0(0) o400->b1211cf5-fced-b5d5-1f8a-9d542b4cf0ee@192.168.124.146@o2ib:0/0 lens 224/0 e 1 to 0 dl 1405539824 ref 2 fl Complete:H/c0/ffffffff rc 0/-1
Jul 16 12:43:55 iws29 kernel: Lustre: 7598:0:(service.c:1308:ptlrpc_at_send_early_reply()) Skipped 1180 previous similar messages
Jul 16 12:44:45 iws29 kernel: Lustre: lustre-OST000a: Client b1211cf5-fced-b5d5-1f8a-9d542b4cf0ee (at 192.168.124.146@o2ib) reconnecting, waiting for 258 clients in recovery for 1:21
Jul 16 12:44:45 iws29 kernel: Lustre: Skipped 1222 previous similar messages
Jul 16 12:45:01 iws29 kernel: program snmpd is using a deprecated SCSI ioctl, please convert it to SG_IO
Jul 16 12:46:42 iws29 kernel: LustreError: 8045:0:(ldlm_lib.c:2302:target_queue_recovery_request()) @@@ dropping resent queued req  req@ffff8805253d5c00 x1473702339243048/t0(4320363938) o4->e8eb0472-0cc1-cdc5-7b35-4137d7e8cbd5@192.168.123.243@o2ib:0/0 lens 488/0 e 0 to 0 dl 1405540063 ref 2 fl Interpret:/6/ffffffff rc 0/-1
Jul 16 12:46:42 iws29 kernel: LustreError: 8045:0:(ldlm_lib.c:2302:target_queue_recovery_request()) Skipped 54 previous similar messages
Jul 16 12:47:07 iws29 kernel: LustreError: 8118:0:(tgt_handler.c:1980:tgt_brw_write()) lustre-OST000a: Dropping timed-out write from 12345-192.168.124.78@o2ib because locking object 0x0:11272550 took 106 seconds (limit was 45).
Jul 16 12:47:07 iws29 kernel: LustreError: 8118:0:(tgt_handler.c:1980:tgt_brw_write()) Skipped 58 previous similar messages
Jul 16 12:47:07 iws29 kernel: Lustre: lustre-OST000a: Bulk IO write error with c43f355b-28da-58ec-e885-107b4a81a73d (at 192.168.124.78@o2ib), client will retry: rc -110
Jul 16 12:47:07 iws29 kernel: Lustre: Skipped 58 previous similar messages
Jul 16 12:50:02 iws29 kernel: program snmpd is using a deprecated SCSI ioctl, please convert it to SG_IO
Jul 16 12:52:32 iws29 kernel: Lustre: ost: This server is not able to keep up with request traffic (cpu-bound).
Jul 16 12:52:32 iws29 kernel: Lustre: Skipped 1135 previous similar messages
Jul 16 12:52:32 iws29 kernel: Lustre: 7536:0:(service.c:1511:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=0, svcEst=45, delay=0(jiff)
Jul 16 12:52:32 iws29 kernel: Lustre: 7536:0:(service.c:1511:ptlrpc_at_check_timed()) Skipped 1135 previous similar messages
Jul 16 12:52:32 iws29 kernel: Lustre: 7536:0:(service.c:1308:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-11s), not sending early reply. Consider increasing at_early_margin (5)?  req@ffff8804ba48b800 x1473702816921736/t0(0) o400->8be9665a-cf85-e197-41db-8001d97a7eef@192.168.125.8@o2ib:0/0 lens 224/0 e 1 to 0 dl 1405540341 ref 2 fl Complete:H/c0/ffffffff rc 0/-1
Jul 16 12:52:32 iws29 kernel: Lustre: 7536:0:(service.c:1308:ptlrpc_at_send_early_reply()) Skipped 2470 previous similar messages
Jul 16 12:53:22 iws29 kernel: Lustre: lustre-OST0016: Client 8be9665a-cf85-e197-41db-8001d97a7eef (at 192.168.125.8@o2ib) reconnecting, waiting for 258 clients in recovery for 1:21
Jul 16 12:53:22 iws29 kernel: Lustre: Skipped 2558 previous similar messages
Jul 16 12:55:01 iws29 kernel: program snmpd is using a deprecated SCSI ioctl, please convert it to SG_IO
Jul 16 12:55:34 iws29 kernel: LustreError: 7684:0:(ldlm_lib.c:2302:target_queue_recovery_request()) @@@ dropping resent queued req  req@ffff880520bfcc00 x1473717364204928/t0(4320364077) o4->5caf10ee-c3ea-8f73-8c45-4a7b1f3740d8@192.168.124.96@o2ib:0/0 lens 488/0 e 0 to 0 dl 1405540595 ref 2 fl Interpret:/6/ffffffff rc 0/-1
Jul 16 12:55:34 iws29 kernel: LustreError: 7684:0:(ldlm_lib.c:2302:target_queue_recovery_request()) Skipped 73 previous similar messages
Jul 16 12:55:41 iws29 LDAPOTP-AUTH[8310]: root@iws1.llnl.gov as root: cmd='uname -r'
Jul 16 12:55:42 iws29 LDAPOTP-AUTH[8351]: root@iws1.llnl.gov as root: cmd='uptime'
Jul 16 12:55:42 iws29 LDAPOTP-AUTH[8392]: root@iws1.llnl.gov as root: cmd='rm -f /etc/badnode;/admin/scripts/nodeup'
Jul 16 12:55:42 iws29 LDAPOTP-AUTH[8452]: root@iws1.llnl.gov as root: cmd='/home/white215/set_max_sect.sh'
Jul 16 12:55:43 iws29 LDAPOTP-AUTH[8495]: root@iws1.llnl.gov as root: cmd='umount -f /p/iws28_1;sleep 10;umount -f /p/iws28_1'
Jul 16 12:55:43 iws29 kernel: LustreError: 8535:0:(ldlm_lib.c:2117:target_stop_recovery_thread()) lustre-OST000a: Aborting recovery
Jul 16 12:55:43 iws29 kernel: Lustre: 8118:0:(ldlm_lib.c:1796:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Jul 16 12:55:43 iws29 kernel: Lustre: 8118:0:(ldlm_lib.c:1442:abort_req_replay_queue()) @@@ aborted:  req@ffff880507108000 x1473702935645796/t0(4320571515) o4->c43f355b-28da-58ec-e885-107b4a81a73d@192.168.124.78@o2ib:0/0 lens 488/0 e 1 to 0 dl 1405540498 ref 1 fl Complete:/4/ffffffff rc 0/-1
Jul 16 12:55:43 iws29 kernel: LustreError: 8118:0:(ldlm_lib.c:1463:abort_lock_replay_queue()) @@@ aborted:  req@ffff880d900ae000 x1473717329952380/t0(0) o101->8787eec0-f320-9f53-5705-9be90adb1d87@192.168.124.182@o2ib:0/0 lens 328/0 e 1 to 0 dl 1405539297 ref 1 fl Complete:/40/ffffffff rc 0/-1
Jul 16 12:55:43 iws29 kernel: Lustre: lustre-OST000a: Not available for connect from 192.168.119.13@o2ib (stopping)
Jul 16 12:55:43 iws29 kernel: Lustre: Skipped 1 previous similar message
Jul 16 12:55:43 iws29 kernel: Lustre: 8118:0:(ldlm_lib.c:2071:target_recovery_thread()) too long recovery - read logs
Jul 16 12:55:43 iws29 kernel: Lustre: lustre-OST000a: Recovery over after 21:35, of 258 clients 0 recovered and 258 were evicted.
Jul 16 12:55:43 iws29 kernel: LustreError: dumping log to /tmp/lustre-log.1405540543.8118
Jul 16 12:55:45 iws29 kernel: LustreError: 8181:0:(tgt_handler.c:1980:tgt_brw_write()) lustre-OST0016: Dropping timed-out write from 12345-192.168.124.180@o2ib because locking object 0x0:11293329 took 106 seconds (limit was 45).
Jul 16 12:55:45 iws29 kernel: LustreError: 8181:0:(tgt_handler.c:1980:tgt_brw_write()) Skipped 120 previous similar messages
Jul 16 12:55:46 iws29 kernel: Lustre: lustre-OST0016: Bulk IO write error with ba19dba5-6c87-f2fe-1a98-2d09fe2c0cbd (at 192.168.124.180@o2ib), client will retry: rc -110
Jul 16 12:55:46 iws29 kernel: Lustre: Skipped 120 previous similar messages
Jul 16 12:55:49 iws29 kernel: LustreError: 137-5: lustre-OST000a_UUID: not available for connect from 192.168.124.35@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jul 16 12:55:49 iws29 kernel: LustreError: Skipped 11 previous similar messages
Jul 16 12:55:49 iws29 kernel: Lustre: server umount lustre-OST000a complete
Jul 16 12:55:57 iws29 kernel: LustreError: 137-5: lustre-OST000a_UUID: not available for connect from 192.168.124.177@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jul 16 12:55:57 iws29 kernel: LustreError: Skipped 119 previous similar messages
Jul 16 12:56:01 iws29 LDAPOTP-AUTH[8542]: root@iws1.llnl.gov as root: cmd='umount -f /p/iws28_2;sleep 10;umount -f /p/iws28_2'
Jul 16 12:56:01 iws29 kernel: LustreError: 8582:0:(ldlm_lib.c:2117:target_stop_recovery_thread()) lustre-OST0016: Aborting recovery
Jul 16 12:56:01 iws29 kernel: Lustre: 8181:0:(ldlm_lib.c:1796:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Jul 16 12:56:01 iws29 kernel: Lustre: 8181:0:(ldlm_lib.c:1796:target_recovery_overseer()) Skipped 1 previous similar message
Jul 16 12:56:01 iws29 kernel: Lustre: 8181:0:(ldlm_lib.c:1442:abort_req_replay_queue()) @@@ aborted:  req@ffff880d2349a800 x1473702059701100/t0(4320364102) o4->e41b94a0-41d9-eb7d-1fc3-0726af6dd09f@192.168.118.129@o2ib:0/0 lens 488/0 e 1 to 0 dl 1405540484 ref 1 fl Complete:/4/ffffffff rc 0/-1
Jul 16 12:56:01 iws29 kernel: Lustre: 8181:0:(ldlm_lib.c:1442:abort_req_replay_queue()) Skipped 15 previous similar messages
Jul 16 12:56:01 iws29 kernel: LustreError: 8181:0:(ldlm_lib.c:1463:abort_lock_replay_queue()) @@@ aborted:  req@ffff88052f0d7400 x1473702660772760/t0(0) o101->ec8bbc42-9e01-af81-a96a-6fb57e969816@192.168.124.183@o2ib:0/0 lens 328/0 e 1 to 0 dl 1405539347 ref 1 fl Complete:/40/ffffffff rc 0/-1
Jul 16 12:56:01 iws29 kernel: Lustre: lustre-OST0016: Not available for connect from 192.168.118.112@o2ib (stopping)
Jul 16 12:56:01 iws29 kernel: Lustre: Skipped 27 previous similar messages
Jul 16 12:56:01 iws29 kernel: LustreError: 8181:0:(ldlm_lib.c:1463:abort_lock_replay_queue()) Skipped 125 previous similar messages
Jul 16 12:56:02 iws29 kernel: Lustre: 8181:0:(ldlm_lib.c:2071:target_recovery_thread()) too long recovery - read logs
Jul 16 12:56:02 iws29 kernel: Lustre: lustre-OST0016: Recovery over after 21:04, of 258 clients 0 recovered and 258 were evicted.
Jul 16 12:56:02 iws29 kernel: LustreError: dumping log to /tmp/lustre-log.1405540562.8181
Jul 16 12:56:08 iws29 kernel: Lustre: server umount lustre-OST0016 complete
Jul 16 12:56:13 iws29 kernel: LustreError: 137-5: lustre-OST0016_UUID: not available for connect from 192.168.118.139@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server.
Jul 16 12:56:13 iws29 kernel: LustreError: Skipped 23 previous similar messages
Jul 16 12:56:19 iws29 LDAPOTP-AUTH[8589]: root@iws1.llnl.gov as root: cmd='umount -f /p/iws28_3;sleep 10;umount -f /p/iws28_3'
Jul 16 12:56:19 iws29 kernel: Lustre: lustre-OST0022: Not available for connect from 192.168.124.134@o2ib (stopping)
Jul 16 12:56:19 iws29 kernel: Lustre: Skipped 24 previous similar messages
Jul 16 12:56:21 iws29 kernel: Lustre: lustre-OST0022: Not available for connect from 192.168.124.169@o2ib (stopping)
Jul 16 12:56:21 iws29 kernel: Lustre: Skipped 73 previous similar messages
Jul 16 12:56:25 iws29 kernel: Lustre: server umount lustre-OST0022 complete
Jul 16 12:56:36 iws29 LDAPOTP-AUTH[8633]: root@iws1.llnl.gov as root: cmd='lctl dl'
Jul 16 13:00:02 iws29 kernel: program snmpd is using a deprecated SCSI ioctl, please convert it to SG_IO
Jul 16 13:05:01 iws29 kernel: program snmpd is using a deprecated SCSI ioctl, please convert it to SG_IO
Jul 16 13:05:01 iws29 LDAPOTP-AUTH[8695]: root@eiws1 as root: cmd='/usr/bin/pdcp -p -y -z /etc'
Jul 16 13:09:23 iws29 LDAPOTP-AUTH[8736]: ROOT LOGIN ON pts/0 FROM iws1.llnl.gov
Jul 16 13:10:02 iws29 kernel: program snmpd is using a deprecated SCSI ioctl, please convert it to SG_IO
Jul 16 13:15:01 iws29 kernel: program snmpd is using a deprecated SCSI ioctl, please convert it to SG_IO
Jul 16 13:20:01 iws29 kernel: program snmpd is using a deprecated SCSI ioctl, please convert it to SG_IO
Jul 16 13:25:01 iws29 kernel: program snmpd is using a deprecated SCSI ioctl, please convert it to SG_IO
Jul 16 13:30:01 iws29 kernel: program snmpd is using a deprecated SCSI ioctl, please convert it to SG_IO
Jul 16 13:35:02 iws29 kernel: program snmpd is using a deprecated SCSI ioctl, please convert it to SG_IO
Jul 16 13:40:01 iws29 kernel: program snmpd is using a deprecated SCSI ioctl, please convert it to SG_IO